H sky from Group 6 20030923
In the absence of any real WFCAM data, we cannot say how that data will be affected (if at all) by any of these. The first two are very instrument and telescope specific. UKIRT's current imager (UFTI) does suffer from both of these and it can be instructive to look at various means of removing them from UFTI data.
The third is site specific and if it exists will be the most problematic. The irony is that the combined temporal/spatial variation of the sky background over the WFCAM field of view is the one effect that we are unable to test. UFTI has such a small field of view that it only covers 1% of the total area of a single WFCAM detector and hence we are unable to measure the variation over the spatial scales that we need.
In a previous paper we reduced a set of UFTI service data. For sky removal we simply combined all images from a given jitter sequence and subtracted the normalised result. Although the jitter-combined images showed no signs of real background residuals, we did not examine the sky subtraction on individual frames in any detail. With this in mind we have revisited this issue using a variant of our normal defringing algorithm.
The data used in the following tests were the nights 20030923, 20031008 and 20031120 on which whole nights of service data were taken to be used for flat field and sky subtraction algorithm testing. 20030923 is a night of data taken in the H band only. The other two were both nights of K band observations. The nights each consist of at least four sets of exposures of test regions, with as many as 50 jitter positions in some cases.
For the sky subtraction tests we first need to get the data to the point where sky subtraction is necessary. The following steps were used in getting the data to this level:
The main structures in the background in the H band are two sets of well
defined fringes. One set is fairly broad and feature mainly in the lower two
quadrants. The other set appear as cocentric rings centred in the upper
right hand quadrant.
H sky from Group 6 20030923
The K band sky exposures also have the cocentric ring structure (now centred somewhere in the lower right hand quadrant). What is different here from the H band is the huge number of slightly defocused sources, which are caused by thermal emission from dust on the cryostat window.
K sky from Group 99 20031003
In a previous paper I demonstrated that removing this sort of structure in non-crowded regions could be accomplished very satisfactorily by a simple subtraction of the "mean" sky frame from a particular jitter sequence, from each frame in that jitter. Difficulties can arise, however, for several reason:
We need to look if it is possible to use sky data from one part of the night to correct for background features in another part of the night. If so, this would solve both above problems,
To address this and the question of the variation of the background components
over the
course of the night, I first combined all of the mean sky frames for a
particular night into one master background frame (this was done with a
suitable weighting scheme to compensate for the fact that some of the mean sky
frames had much better signal/noise than others). I then scaled this master
sky frame to each of the mean sky frames. This was done using our standard
fringe scaling algorithm which attempts to robustly minimise the mean
absolute deviation of the background of a defringed image.
Once the scale factor was found it was
a simple matter of scaling the master fringe frame by that factor and
subtracting it off. Any residual features should then be obvious.
Considering the H data first (night 20030923), below are the four mean sky
frames before and after defringing with the master sky file.
H Sky Group 6
H Sky Group 69
H Sky Group 177
H Sky Group 236
In all four cases the two main fringe patterns are removed completely. Two significant residuals remain however. The first is a set of fringes that appear sky6 and sky177 and which run nearly vertical. Although these appear to be rather strong it must be pointed out that the maximum peak to trough intensity of the original fringes was between 3 to 4% of sky in these frames. These residual fringes are about 0.2% of sky at the very worst. They are only visible in sky frames 6 and 177, which have a much higher signal to noise ratio that the other two. (Sky 6 and 177 have a total exposure time of 1800s. Sky 69 has 735s and Sky 236 has 125s) In the lower exposure observations these fringes are lost in the sky noise. That these residuals appear to shift across the frame shows some sort of phase difference between them and the main fringe patterns.
The question here is whether it is worth taking any more heroic steps to try and remove these completely, perhaps with some sort of independent component modelling as mentioned above. We have to remind ourselves that what we are looking at here are two frames with a total 1800s of exposure time each. What we are trying to achieve is an optimal way to remove background features from single exposures. Features visible in high signal to noise x-y stacked images (ie. no shifts between images) such as these will be much less prominent in single images and are not generally detectable above sky noise. Furthermore, in stacking the jitter sequence to create astronomical images the implict offsets used cause the fine-grained fringe pattern to disappear completely since the offsets essentially randomize its systematic spatial contribution. Providing the systematics are small compared to the rms photon noise this has negligible impact on the quality of the resulting stacked frames. In this this case, trying to do a second order fringe removal may in fact just increase the sky noise as the scaling factors will have very larger than normal uncertainties.
To illustrate these points let's step back and look at a single frame that has been badly affected by fringing.
H observation 16 after dark correction and flat fielding, but before defringing
H observation 16 after defringing with a sky formed from all observations in group 6 (to which this observation belongs)
H observation 16 after defringing with a sky formed from all observations in the night
The first attempt at defringing removes the background features completely. (It is reassuring to note that the scale factor that the defringing algorithm found to be optimal for this observation was 0.9988). This tells us that using a sky estimate that is derived from the target observations themselves is probably the way to go if we have that option. The second attempt at defringing using a nightly mean sky estimate does still leave small residuals. In this particular frame, the peak to trough difference is about 1.2 counts out of a a mean background of 1037.0, or roughly 0.1%. If this frame were to be used with a set of similarly corrected images to form a dither any such residual structure would be washed out during the dithering stage. Given that there may be situations where you cannot use observations of crowded regions to define a mean sky frame, this shows that it is still possible to get perfectly acceptable results using a nightly mean sky.
The other significant residual is a sort of rippling effect that can be seen in all four defringed mean sky frames which is nearly horizontal. This is a result of using a single dark frame to estimate the dark current and reset anomaly. Below is dark frame f20030923_00235. The ripples are clearly seen and are obviously the cause of the effect seen in the sky frames. This will only be sorted out when a proper dark subtraction is done using many averaged dark frames.
Dark frame for group 235
A dark frame created by averaging nine 40 second dark exposures
K Sky Group 99
K Sky Group 153
K Sky Group 297
K Sky Group 382
Using the defringing algorithm to remove both fringes and thermal emission features leave some residual structure on most of the frames. This is a demonstration of variations in these features that are out of phase with each other. This is not at all surprising as they have completely different physical causes. Sky 9 shows a positive residual dust feature in the upper right hand quadrant, that is lying on a negative fringe residual. The opposite is true of Sky 297. Although again these look bad, the worst peak to trough residual fringe in any of these is less that 0.25% of sky and affect only the very high signal to noise frames in a significant way. In the event that a user wanted to do a second order correction to remove more of the residual features, the fact that the features seem to divide so neatly into just two components would ensure that probably only one further correction would be needed.
Let's step back again and look at a single image as we did in the previous section. Below are images of a K band image that has been badly affected by dust and fringing.
K observation 99 after dark correction and flat fielding, but before defringing
K observation 99 after defringing with a sky formed from all observations in group 99 (to which this observation belongs)
K observation 99 after defringing with a sky formed from all observations in the night
Again, using the nightly mean sky estimate does cause some small residual fringes to be visible in the third image. These are barely detectable however and amount to less than 0.08% of sky. This again tells us that in cases where we cannot use the local observations for a sky estimate, that a nightly mean sky can be used for very little cost. It is interesting that although there appears to be residuals in the fringes, this is not the case for the dust emission. This is likely to be because the dust emission is a much greater contribution to the total background emission to be subtracted. If there is some sort of temporal variation in either effect, then the algorithm is most likely to remove the dust better than the fringes.
K Sky Group 99 from 20031008 and K Sky Group 41 from 20031120