Notes from UKIDSS meeting 25th November 2002 Updated after CASU-WFAU liaison meeting 9th December 2002 (+NCH, IB and SJW)
Updated with new timescales 18th March.
Overall philosophy
------------------
Summit - causal pipeline: DQC feedback, first pass science products
Standard - nightly pipeline: DQC check, astrometric and initial
photometric calibration, standard catalogue parameters,
final 2D science images
Further - nightly pipeline: PSF estimation and general profile fitting
Database - asynchronous processing: stacking, difference imaging,
driven temporal, matching and colocated list driven stuff
(inc. external list drivers), contiguous tiled regions and
seamless overlap calibrated catalogues, optimal detection and
parameterisation of eg. LSBs (ie. bigger than PSFs)
(note basic --> standard since not such negative connotations and
advanced --> Database driven since some of the processing not that
advanced per se)
2D instrumental signature removal
---------------------------------
effects to deal with may include: persistence (if present), cross-talk,
dark stability and non-linearity, flatfield stability, defringing,
sky variation, telescope/dust thermal emission
non-linearity, reset-correction (de-biassing) dealt with in data
aquisition system
Issues:
variable seeing on short timescales, high proportion of bad pixels
should also consider stacking cf. interleaving for shallow surveys ?
may need to do PSF fitting on individual components of interleave
irrespective of above
for shallow surveys can we use 16 bit unsigned products rather than -32 bit
reals ? can do for much of pipeline anyway
Test data:
need to aquire test data from WFCAM to assess persistence, reset anomalies,
dark stability, flat stability, bad pixel problems etc... asap
what other UKIRT?WFCAM test data do we need ? - more flats (dome cf.
twilight sky cf. dark sky) dark stability, WFCAM test data - cross-talk
image persistence, reset anomaly, dark stability
Standard 2D processing at summit and in Cambridge to be ready for Version 1.
with some tuning during commissioning and science verification.
Standard products
-----------------
Additional image parameters:
Kron radius: r_k equal to 1st moment of radial distribution
Petrosian radius: r_p defined in terms of annular surface brightness
parameter \eta
Kron flux: summation out to k*r_k can be elliptical or circular ?
(similar to the total flux which it could replace)
Petrosian flux: summation out to k*r_p only seen circular, is elliptical
worth having ?
FWHM alternative: image FWHM summation out to k*FWHM, since for standard profiles is at least as good as above two finely? sampled circular aperture fluxes of fixed sizes - similar to what we already have except more
Pk, core/2, core/sqrt(2), core, sqrt(2)*core, 2*core,
2*sqrt(2)*core, 4*core, 4*sqrt(2)*core,
8*core, 8*sqrt(2)*core,
16*core, 16*sqrt(2)*core, 32*core
where core = ~for stellar images for site ie. ~0.5 arcsec Note that if we set core = 0.5 arcsec then the above sequence leaving out Pk, would read apertures of diameter:
which makes for a pleasing spacing. [see Staunton etal. p 498 for SDSS apertures - problem with them is they are too coarse a grid (eg. x2.5 steps in radius) and extend to far too large a radius given the likely 2D processing constraints on WFCAM data] Note that for circular apertures all of Kron, Petrosian and FWHM are subsumed within circular aperture fluxes if interpolate from fine enough grid, as are SDSS apertures (different seeing etc.. implies will need to interpolate/correct to really match the SDSS ones anyway) How fine ? what spacing ? how big a radius ? above are suggested set. finer grid means can always pick optimal aperture for stellar images aperture corrections for stellar images are useful adjunct (already computed) Note that standard background estimation ultimately defines max. radius as may 2D processing and is expected to be <~30 arcsec. couldn't see anything extra in Sextractor list worth having not covered in above and current lists Image morphological classification: is spurion, stellar, non-stellar enough together with the pseudo N(0,1) statistic ?, could use Bayesian priors when calibrated + other knowledge eg. globular cluster..........
and is mainly based on the curve-of-growth defined by the aperture fluxes,
profile fit results.
Image sizes and ellipciticy, are useful extra
information thrown into the image classification pot.
The Bayesian prior
knowledge will be needed to produce sensible P(star), P(galaxy) using a
standard Bayesian classification scheme.
need to add in errors on certain parameters and processing flag a la SDSS one eg. set bit for hit edge etc...
Standard products to be ready for Version 1. with some tuning during
commissioning and science verification.
Further products
----------------
PSF generation methods, oversampled spatially varying PSFs good news,
how to form them cf. SDSS, HST, 2MASS ?
may need to use components of interleave ? and for other possible 2Ds fits
such as Sersic.
oversampled real PSFs vital for correct fitting of undersampled images
Options for PSF fitting are:
1) Profile fit without centroid adjustment using standard multiple components
2) Profile fit with centroid adjustment using standard multiple components
3) DAOPHOT/CLEAN algorithm (Full Monty) - multiple sources with full iterative
solution, adding and removing components as necessary (CPU intensive) and
obviously won't work for galaxies ie. prbably useless for generic higher
latutidue fields
do we additionally need more general profile fits eg. de Vaucouleur,
exponential, Plummer ???
Aligning with DB versioning: further processing Version 1. and 2. would include options 1,2 for PSF work. Option 3 would be developed for Version 3. 1D/2D Sersic fits would be developed for Version 2. Database-driven products ------------------------ Organised by DB but run using general purpose selectable agorithmic toolkit asynchronous operations such as list driven photometry, vital to make same measurements across all bands master list created how? from union of catalogues in each band ? + external lists ? stacking images first then generating catalogue in general not so good since s:n will be worse for interesting images, should be RA, Dec list list drives software processing via WCS, can use object masks to improve standard background estimation. External (no-WFCAM) lists can also be incorporated. Version 1. just WFCAM data Version 2. externally derived lists Version 3. cross-photometry on external datasets stacking, complex problem, not necessarily single optimum solution eg. image detection cf. resolution of higher s:n images, Subaru involvement, JPE's e-science project, loads of possible methods, inc. re-running image detection alogrithms
Version 1. stacker and mosaicer, straighforward algorithms with ability to
deal correctly with, scaling, confidence maps, spurion rejection, and
accurate WCS alignment
Version 2. optimal use of varying PSF information for image detection
problem and for image resolution problem.
Version 3. complete image restoration using all data contemporaneoulsy
LSBs, or more generic image detection methodology, not discussed at
UKIDSS meeting, is it worth pursuing for NIR arrays ?
Version 2. re-detections using fixed large detection filters having
previously removed standard image detections. Assess feasibility of
LSB work in NIR mosaic data.
Version 3. await results for Version 2. before proceeding.
detection of transient phenomena, adpative kernel matching, difference
imaging continuum subtraction, inc. re-running image detection alogrithms
Version 1. ability to do continuum subtraction for narrow band (eg. H2) work
Version 2. more sophisticated difference imaging using adaptive kernel
matching
Version 3. the rest
--------------------------------------------------------------------------
Timescales for Versions 1-3 are:
Version 1. ready for WFCAM commissioning eg. end of 2003
Version 2. ready for one year after survey operations begin eg. notionally
end of 2004
Version 3. rollout sometime later though development can be concurrent
--------------------------------------------------------------------------
Other Issues
------------
Setting up standard photometric calibration fields regularly spaced in RA
need to ensure this gets done early on and that standards in all broadbands
used in a night are taken at regular (1hr?) intervals. STH to coordinate.
Skyprobe local extinction monitor -v- ad hoc mishmash of AG measures etc..
MJI, DWE, STH, JRL
-------------------------------------------------------------------------
ACTIONS
-------
1. CASU to produce updated CDR document taking account of recent
discussions - needs timeline for various components ONGOING
2. WFAU to produce updated document ditto, both by end of
December DONE
3. MJI to contact MMC about acquiring recent WFCAM test data,
(received test data and report) DONE
4. STH Dome flats -v- twilight flats -v- dark sky flats, test data
still needed, STH to try again in December - nada. AA to
attempt with UFTI on 15th January ONGOING
5. JRL to get prototype WFCAM FITS header for images and forward to
WFAU for comments ONGOING
6. JRL to put together example data and catalogues from CIRSI data
for use in tests of ingestion, stacking/mosaicing at WFAU ONGOING
7. SJW to contact MMC about commissioning duration and plans,
particularly with regard to setting up photometric standard fields ONGOING
8. STH assuming CASU processes the data who is going to do colour
equations produce the standards and (keep updating them) do meso STH
slew tests VOLUNTEERED
9. MJI to ask MMC about plans for DAS 5s exposures, 16 bit alternative
data format DEBATING
10. SJW to set up simulations to test parameter generation software,
sensitivity to seeing etc... ONGOING
11. SJW to provide/select a night of UFTI/UIST data to compare 2D
processing ONGOING
12. WFAU to do more tests to find bottleneck in network connectivity
between Cambridge and Edinburgh. Current large scale transfer
achieved speed at 2.4 Mbit/s is hopeless ONGOING
13. CASU to forward prototype stacking and mosaicing software to WFAU for
tests. DONE
Last modified: Wed Mar 19 10:52:34 2003