I did some experiments on compression using CIRSI data. The first lot were raw CIRSI images stored as 16 bit integers. The second were completed dithers which were stored as 32 bit reals. I tested 4 methods for compression ratio and speed. Unfortunately the speed was not something I was able to measure in a quantitative way. I did get a good qualitative feel for the speed, though. The four methods were: (1) Tile compression with the Rice algorithm (lossy for real data only) (2) Tile compression with the gzip algorithm (lossy for real data only) (3) gzip on a whole file (lossless) (4) bzip2 on a whole file (lossless) The 'tile compression' is a method that is native to cfitsio and not much else at present. The algorithm is lossless for integer data. For real data the user can tune the lossiness with the 'noisebits' parameter. There is obviously a balancing act between preserving the data and getting a decent compression ratio. In the results below I present a value called 'sigma diff'. This is the sigma of the difference between the original frame and an uncompressed frame that has had a lossy compression applied and is some sort of measure of the lossiness. Tile Rice algorithm: -------------------- completely raw cirsi images (original images 16 bit integers). mean compression factor 1.85, no loss dithers (original images 32 bit real) mean compression 4.77 for noisebits=4, sigma diff = 0.110 mean compression 3.67 for noisebits=6, sigma diff = 0.027 mean compression 2.99 for noisebits=8, sigma diff = 0.007 masking out bad pixels in original images increases compression by negligible amount. Tile GZIP algorithm: -------------------- completely raw cirsi images mean compression 1.25 dithers mean compression 3.56 for noisebits=4, sigma diff = 0.114 mean compression 2.58 for noisebits=4, sigma diff = 0.028 mean compression 2.18 for noisebits=4, sigma diff = 0.007 Gzip on whole file: ------------------- completely raw cirsi images compression factor 1.47 dithers compression factor 1.40 BZIP2 on whole file: -------------------- completely raw cirsi images compression factor 1.87 dithers compression factor 1.78 Timing: ------- Gzip (both tile and whole file) and bzip2 take noticably much more time than Rice. I can't be more specific that this as the Rice compression doesn't even give me a chance to hit the button on my stopwatch. Community Support ----------------- Here is the response of Eric Mandel (SAO) with regards to support for tile compression in DS9: ------------------------------------------------------------------------------- Dear Jim, I am not familiar with "tile compression", which I guess is a cfitsio convention. We generally try to support the FITS standard and also a few extensions for which there is widespread community demand. As "tile compression" is in neither category at the moment, we have no plans to add it to ds9. Regards, Eric ------------------------------------------------------------------------------- **** NOTE added by MJI - we have already had need to modify DS9 to accept ZPX WFS and could if needed create our own version that reads compressed FITS on-the-fly. ------------------------------------------------------------------------------- and from Nelson Zarate (IRAF): ------------------------------------------------------------------------------- Hello Jim; I am working on a prototype at this time so that IRAF can read and create compress type using the RICE compression algorithm. The syntax to create a FITS compress extension is something like: imcopy file.fits filec.fits[type=compress] Notice that we do not put tile size at this time and the iraf default would be row by row compression. I'd like to put one restriction to this new IRAF image support and that is to not open an image in READ-WRITE mode. My release date is around July with a MORE OR LESS added. Cheers, Nelson ------------------------------------------------------------------------------- Clearly there is some support for this, although perhaps not as much as we would like. From the point of view of JAC, CASU and WFAU, it might be judicious for us to use this format in order to cut down data transfer and storage requirements. It would be a very simple matter for WFAU to convert any compressed FITS images after copying. If we are going to move to 5s exposures for standard survey observing with WFCAM, this will increase the original estimated data rate to an average of 300 Gb per 10 hour night (32 bit real data). Using the tile Rice algorithm and insisting on 16 bit integer data would reduce this by a factor of ~4 to roughly 75 Gb per night.