I did some experiments on compression using CIRSI data.  The first lot were
raw CIRSI images stored as 16 bit integers.  The second were completed dithers
which were stored as 32 bit reals.  I tested 4 methods for compression ratio
and speed.  Unfortunately the speed was not something I was able to measure
in a quantitative way.  I did get a good qualitative feel for the speed, 
though.  The four methods were:

(1) Tile compression with the Rice algorithm (lossy for real data only)
(2) Tile compression with the gzip algorithm (lossy for real data only)
(3) gzip on a whole file (lossless)
(4) bzip2 on a whole file (lossless)

The 'tile compression' is a method that is native to cfitsio and not much else
at present.  The algorithm is lossless for integer data.  For real data the
user can tune the lossiness with the 'noisebits' parameter.  There is obviously
a balancing act between preserving the data and getting a decent compression
ratio. In the results below I present a value called 'sigma diff'. This is
the sigma of the difference between the original frame and an uncompressed
frame that has had a lossy compression applied and is some sort of measure 
of the lossiness.

Tile Rice algorithm:
--------------------
completely raw cirsi images (original images 16 bit integers).
	mean compression factor 1.85, no loss

dithers (original images 32 bit real)
	mean compression 4.77 for noisebits=4, sigma diff = 0.110
        mean compression 3.67 for noisebits=6, sigma diff = 0.027
	mean compression 2.99 for noisebits=8, sigma diff = 0.007

masking out bad pixels in original images increases compression by 
negligible amount.

Tile GZIP algorithm:
--------------------
completely raw cirsi images
	mean compression 1.25

dithers
	mean compression 3.56 for noisebits=4, sigma diff = 0.114
	mean compression 2.58 for noisebits=4, sigma diff = 0.028
	mean compression 2.18 for noisebits=4, sigma diff = 0.007


Gzip on whole file:
-------------------
completely raw cirsi images
	compression factor 1.47

dithers
	compression factor 1.40

BZIP2 on whole file:
--------------------
completely raw cirsi images
	compression factor 1.87

dithers
	compression factor 1.78


Timing:
-------

Gzip (both tile and whole file) and bzip2 take noticably much more time than 
Rice. I can't be more specific that this as the Rice compression doesn't even
give me a chance to hit the button on my stopwatch.

Community Support
-----------------

Here is the response of Eric Mandel (SAO) with regards to support for tile
compression in DS9:

-------------------------------------------------------------------------------

Dear Jim,

I am not familiar with "tile compression", which I guess is a cfitsio
convention. We generally try to support the FITS standard and also a
few extensions for which there is widespread community demand. As
"tile compression" is in neither category at the moment, we have no
plans to add it to ds9.

Regards,

Eric

-------------------------------------------------------------------------------

**** NOTE added by MJI - we have already had need to modify DS9 to accept
          ZPX WFS and could if needed create our own version that reads
          compressed FITS on-the-fly.

-------------------------------------------------------------------------------

and from Nelson Zarate (IRAF):

-------------------------------------------------------------------------------

Hello Jim;
        
        I am working on a prototype at this time so that IRAF can read
and create compress type using the RICE compression algorithm.

        The syntax to create a FITS compress extension is something like:

        imcopy file.fits filec.fits[type=compress]

     Notice that we do not put tile size at this time and the iraf default
would be row by row compression.

     I'd like to put one restriction to this new IRAF image support and that
is to not open an image in READ-WRITE mode.

      My release date is around July with a MORE OR LESS added.

Cheers,
        Nelson

-------------------------------------------------------------------------------

Clearly there is some support for this, although perhaps not as much as we
would like.  From the point of view of JAC, CASU and WFAU, it might be 
judicious for us to use this format in order to cut down data transfer and
storage requirements.  It would be a very simple matter for WFAU to convert
any compressed FITS images after copying.

If we are going to move to 5s exposures for standard survey observing with
WFCAM, this will increase the original estimated data rate to
an average of 300 Gb per 10 hour night (32 bit real data).  Using the tile 
Rice algorithm and insisting on 16 bit integer data would reduce this by a 
factor of ~4 to roughly 75 Gb per night.