A  WFCAM Pipeline Science Requirements

This list combines all requirements from PDR, with additional input from JAC and UKIDSS. It describes the expected pipeline inputs (by defining the standard observing strategies which the pipeline will have to accommodate), the specific summit feedback parameters which will be used to guide observing in pseudo-real time, and the details of the final output from the Cambridge pipeline.

A.1  Top-level requirements

T1 The Summit and Cambridge pipelines must be functionally identical, produce identical output given the same input and the same processing steps, be based on the ORAC-DR recipe/primitive philosophy (and therefore be configurable at both the primitive and recipe level).
T2 The pipeline should be able to function in the absence, temporary or semi-permanent, of (data from) one or more WFCAM detector arrays.
T3 The summit pipeline in particular must be able to make sky-conditions/DQC information available to the OMP Query Tool. [PDR requirement 6.1]
T4 The DR system must be able to complete any processing required fast enough to keep up with data taking. [PDR req. 6.1]
T5 The DR system must be able to provide quick feedback on the quality of the raw data to a user of WFCAM. [PDR req. 6.2].
T6 The DR system must be capable of providing the QCPs (Quality Control Parameters) required to automatically monitor the quality of the data (using WFCAM survey recipes). [PDR req. 6.3 - note that OCDD no longer lists QCP parameters]
T7 The DR system must be capable of comparing the values of QCPs with defined limits, and raising alarms if the values are outside of those limits. [PDR req. 6.5]
T8 The DR system must capable of standalone operation, i.e. of running without supervision. The data reduction recipe to be used must be supplied in the raw data FITS headers. The observing sequence will ensure that suitable calibration observations are available. [PDR req. 6.4]
T9 The DR system must provide logging information to disk. [PDR req. 6.6]
T10 A recipe for automatically determining best focus from a focus run must be provided. [PDR req. 6.7]
T11 Pipeline version number must be recorded in FITS headers of reduced data
T12 Pipeline must provide recipes to reduce MSBs undertaken in nonphotometric conditions, to a point where the reduced frames can be merged with the rest of the survey concerned in the science archive.

A.2  Observing Sequences

The pipeline should be able to fully reduce data from the standard observing sequences noted below. With the choice of 32-channel readout, the minimum exposure time can be reduced; this may open the possibility of dithering to remove bad pixels, so "microstepped sequences" below should be taken as possibly meaning "jittered series of microstepped sequences" pending further discussion (of feasibility and data rate).

A.2.1  Individual Microstepped survey frames

O1 1×1 microstep
O2 2×2 microstep sequence
O3 3×3 microstep sequence

A.2.2  Complete Microstepped survey tiles

O4 Complete tile of four 1×1 microsteps
O5 Complete tile of four 2×2 microsteps
O6 Complete tile of four 3×3 microsteps

A.2.3  Variants

O7 All of modes 1-6 may be repeated immediately in a series of filters. It is not anticipated that filter changes will be made within the space of one microstep sequence, nor that any Observation will consist of Integrations in multiple colours.
O8 All of modes 1-6 may be carried out in narrow-band filters, in which case "on-line" and "continuum" frames will require subtraction

A.2.4  Standard fields

O9 2×2 microstep or microstep/mesostep sequence on a standard field (mesostep is an offset sequence large enough to allow determination of the vignetting function).
In all of modes 1-6, the survey tiles will be supported by observations of suitable standard fields, at appropriate intervals. Standard stars will always be observed in all broadband filters, unless the observing for the night is known to be constrained to a smaller number of filters.

A.2.5  Extended object

O10 Large extended field (complete tile interspersed with jitter frames on a remote sky patch)
O11 Crowded field: 2×2 or 3×3 microstep sequence with equal number of interspersed sky observations

A.3  Summit feedback

The summit pipeline must reliably generate a number of sky-related parameters which will be used to select from the available survey MSBs. These are to be filed as calibrations or otherwise made available to the Query Tool.
F1 Sky brightness and sky noise measurement (per filter)
F2 Mean stellar image diameter (determined by fitting within the processed frames, filed as a calibration after conversion to K-band equivalent).
F3 Instrumental zero point determined from 2MASS stars in the survey processed frames (or the difference between this and the expected value). Stored as a calibration and made available to the QT.
F4 "Locally photometric" flag (determined from variations in detected image properties in the frames in the microstep sequence).
F5 "Globally photometric" flag (determined from all previous observations of standard fields on the current night).
F6 Limiting magnitude (5s) in the processed frame.
F7 Mean axial ratio of suitable stellar images.
F8 Nightly photometric zero point from observations of standard fields

A.4  Summit QC parameters

The summit pipeline must produce the following parameters relating to quality control of the instrument and telescope. Note that these supplement QC monitoring done in the controller and Linux PC. The latter will be in the raw data FITS headers and it is assumed that they will be propagated. Daytime instrument checks will be carried out using the engineering pipeline provided separately by the WFCAM project.
Q1 Limiting magnitude (5s) in the superframe (possibly normalised to one second exposure time), recorded in an index file. A final check for subtle noise problems, used to trigger an alarm message by comparison to a predefined range.
Q2 Mean axial ratio of bright sources (probably those in F4) - a check for astigmatism and trailing, used to trigger an alarm message by comparison to a predefined range.
Q3 Image diameter (determined by fitting within the reduced superframes, filed as a calibration after conversion to K-band equivalent). The calibration value must either be the average of that from the four individual frame pipelines or (more likely given T2) that from one pre-specified array.
Q4 Sky brightness (one value per filter) - determined from mode or median of pixel values. Recorded in FITS headers and used to trigger an alarm by comparison to a predefined range.
Q5 Image periodic noise (looking for, e.g., 60Hz pickup) - to be done within each reduced frame.
Q6 Running log (filed as calibration) of last available photometric zero-point from the last-observed standard field in each filter. Anticipating possible operation with different filters in the four positions in a paddle, all available zeropoints should be included, not just the one appropriate to the current filter.
Q7 Number of bad pixels, determined from last-determined flatfield as the number of pixels lying outside a pre-specified range.
Q8 Number of hot pixels measured from dark frame
Q9 Dark count per second

A.5  Other summit output

D1 Display of intermediate frames should be configurable via the ORAC disp.dat system
D2 Diagnostic messages should appear on the oracdr
D3 Display of complete tiles in the case of O4-O6. Options selected through disp.dat.

A.6  Cambridge and summit pipeline functions

Note: nonlinearity will be dealt with in the acquisition PC and will not be a function required of the DR pipeline.
C1 Removes instrumental signatures: bias, dark, flat field (to precision limited by photon statistics and array systematics), bad and hot pixels, channel variations, scattered light, vignetting correction, fringing correction where possible. If necessary, applies cross-talk matrix to remove interference between channels and arrays. Maintains and uses running calibrations (superflat, bad pixel mask, master dark frame etc.)
C2 Removes (as far as possible) atmospheric OH emission and sky background, other 2-D background variations arising in the detector, and cosmic rays
C3 Removes persistent images from previous exposures
C4 Interleaves microstep frames to produce one superframe per array
C5 Produces and propagates "confidence map" for the superframe, and propagates this into tiles if produced
C6 Combines superframes into a tile if part of a tile observing pattern, and develops a consistent internal calibration scheme to put observations on an approximately uniform system.
C7 Generates source catalogue (after deblending in crowded fields) containing astrometric, photometric, shape and DQC parameters
C8 Flux-calibrates to 2% (limited by Poisson noise), using all-night or running photometric information, superflats etc. as appropriate.
C9 Astrometry on catalogues and reduced pixel images, the latter to use an agreed and appropriate WCS in all FITS headers. External accuracy limited by input catalogue; internal accuracy limited by s/n, detector stability etc. Goals are 250 and 100mas respectively, with an ultimate goal of 20mas for internal systematics.
C10 Measures running atmospheric extinction coefficient (summit pipeline), records in an index file
C11 Determines nightly extinction coefficient (Cambridge pipeline), records in FITS headers
C12 Provides a recipe which resamples and combines a complete tiled image, without generating catalogue (useful for large extended fields, as in O10)
C13 Provides a recipe to determine residual systematics and scattered light functions from a mesostep sequence on a standard field
C14 Provides a recipe to enable determination of best focus from a series of frames taken at different focus positions (focus value will be included in FITS headers of raw data).

A.7  Cambridge pipeline data products

Note: there are a number of possible ways to catalogue in the case of overlapping frames and/or complete tiles. It is assumed that the summit and basic pipeline's jobs are to reduce the data to hand (microstepped superframe or complete tile) without reference to catalogues from frames in overlapping fields taken previously.
Note: anticipated content of FITS files coming from CASU to WFAU is up to three files per array per pointing: one of the superframe pixel data, one of the associated confidence map, one of the associated source catalogue. In the case of tiled observing modes: one of the mosaiced superframe, one of of the associated confidence map, one of the associated catalogues.
P1 FITS image and assoicated confidence map of each processed frame (one per array, N(arrays) per pointing; this applies in observing modes O1 to O3).
P2 FITS line-emission processed frame (in observing mode O8).
P3 Photometric calibration information and confidence maps for each processed frame
P4 Catalogue of objects detected in the processed frames (one per array; this applies in observing modes O1 to O3). Catalogued image properties are detailed elsewhere.
P5 For tiled observing modes, both the products detailed in P1-P4 as appropriate, plus catalogue and confidence maps for the completed tile will be supplied.

A.8  Catalogued Image Properties

I1 This table is either the Sloan list or the Cambridge list, or a merged/truncated version of both. For the summit pipeline the APM set would be adequate; users requiring Sloan-style parameters can wait for the data to be processed in the UK.
I2 Realistic error estimates should accompany all parameters for which this is possible



File translated from TEX by TTH, version 3.30.
On 24 Jan 2003, 15:51.