26th July 2005 Hoyle committee room 11.30 - 13:25 Present: MJI, RGM, PSB, DWE, MR, JRL, WJS, EGS, NAW Apologies: STH Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Data archives update - UKIRT, WFCAM, AAO, ING 4. Optical processing update 5. WFCAM JAC telecons and processing status 6. VDFS update 7. AstroGrid liaison update 8. AOB - ADASS 2005 Minutes ------- 1. Actions from last meeting JRL has still to include access to "observing" log index files in the raw WFCAM archive. It was decided to transfer this action to MR as part of the interface update MJI are still in the process of writing a new report on the detector JRL properties as they currently are and the processing strategy. This MR is needed for the SV meeting in Edinburgh (18 August) and should be finished about a week before this. It was decided that EGS's report <<<< should be separate since it deals with the internal science verification. EGS, MR, MJI and STH are the likely attendees. JRL AAO have not replied yet regarding the connectivity issues to AAO. It was suggested that in order to progress this an additional email (cc'd to various other interested parties) or a telephone call was required - ongoing <<<< WJS was unsure whether the Nov 2004 version was the latest SADT Definition Document. He said that he would find out what is happening - ongoing <<<< MJI raised the issue of observing efficiency at the JAC telecon (see item 5) RGM said that the AstroGrid hardware money was still there RGM sent PSB's comments on the document covering the service support for VO to Tony Linde STH has yet to trial the test version of the SADT available for both VISTA and VST - ongoing <<<< ALL only DWE sent in a Q3 plan. MJI will draw up a quarterly draft plan and circulate it for comments. The finalized plan needs to be done in advance of the next VDMT meeting (11 August) - ongoing <<<< MR has not yet improved the look of the front page of the WFCAM raw data archive - ongoing <<<< JRL is half-way to sorting out local scripts for easy access to the mirrored OMP data base tables - ongoing <<<< MR has investigated Aladin for parts of the CASU WFCAM processed DQC archive. He said that although it was in Java (painful), it would be useful because of its generality. It was decided this should now go into the Q3 plan rather than remaining an action. JRL has run his summit pipeline timing tests on other nights, in particular those containing DXS and GPS observations where the pipeline has to work hardest. On average, the pipeline only gets a 5 minutes backlog when processing this type of data. He will report on this at the next JAC telecon. <<<< MR has investigated the properties of all flats taken during 05A and found them to be globally stable to better than 1% for all filters. The noise properties were also OK. More detail will go in the aforementioned report. JRL was asked to look at the sky calibration frames to see if there is any correlation between Moon angle and scattered light. <<<< DWE has put in a link to the processing progress web page and the raw data archive off the main VDMT web page. MR has started to put some explanatory text in the processing progress web page but has not finished - ongoing <<<< STH has not finished the 3 remaining RIXs in the ETC document. This has to be finished by 5 August - ongoing <<<< PSB forwarded the Michele Peron email to JPE and JMS PSB has checked the status of the various VISTA documents. Not all are up-to-date - ongoing <<<< WJS has not yet checked the VISTA template definitions - ongoing <<<< MJI has not yet raised the calibration/science frame issue at the VDMT since there hasn't been one since the last meeting. The general feeling was that it would be simpler to classify the observations of the standards as calibration frames - ongoing <<<< PSB has contacted JPE about the status of the SADT with respect to the RIXs - ongoing <<<< 2. Comments on WFAU minutes RGM said that he had managed to access the WFAU archive. There was a short discussion regarding Petrosian magnitudes and other flux measures. 3. Data archives update - UKIRT, WFCAM, AAO, ING UKIRT: about 20 more requests have come in. We are about 1 year behind the observations ie. up-to-date with the proprietary period. WFCAM: all the raw data has been ingested. About 2 more weeks worth of data await transfer to ESO. AAO: awaiting resolution of connectivity issue. ING: over the last month data has been delivered over the Internet (~150 Gbyte/month). We are currently keeping up with the observations. This method requires much less effort that reading in DVDs. JRL suggested that Rice compressing this data might be useful. MJI noted that the data was gzipped prior to transfer and suggested that we could wait another month for the dust to settle before starting on this. 4. Optical processing update EGS said that he had been processing INT WFC data for Alan Fitzsimmons and Hans Deeg. There were no problems at the moment in keeping up with this. RGM said that an XMM user wanted to stack the processed products where applicable. MJI said that the simplest solution was to use our stacking software. EGS said that he had been using this for various WFCAM tests and would make this available. <<<< 5. WFCAM JAC telecons and processing status The next JAC telecon will be tomorrow evening. The main topic of discussion will be the pipeline requirements and how far they have been met. Problems with 3x3 interleaving and also the chip 2 dropouts will be rasied. At the last JAC telecon (15 July) the main topic was observing efficiency and the design of the MSBs for the UKIDSS surveys. RGM asked what fraction of time is spent observing standards. This did not seem to be factored into the various survey observing plans since the old calibration plan was to use 2MASS. It was not clear though when this earlier plan was abandoned (2003?) and whether or not the impact of the new calibration strategy had been assessed. NAW asked how would the summit pipeline be affected if the observing strategy for the MSBs were to be changed to be more efficient by using less interleaving, longer exposures or more coadds. It was pointed out that there generally would be less frames to process, the summit processing is quite flexible, and the load on the summit pipeline would therefore generally be less. DWE gave some more details regarding the observing efficiency derived from the timing information in the FITS headers. For each observation (integration) there is about 1.2s of reset-read-read overhead per exposure. If there is a filter change or there is a large movement of the telescope there is an overhead of ~30-45s. The remaining overhead is 2-3s per observation. There doesn't seem to be any correlation between this overhead and the amount of dither movement that has been made. Further more detailed information on the current and expected overheads are being sent out by JAC for discussion at the Survery Heads meeting in Durham. MR said that the reductions were progressing well. Almost all have been finished apart from some frames that need some manual intervention. Data up to 6 June is ready and waiting for transfer to Edinburgh. Transfer of data has been completed up to 10 May. He said that the catalogue generation can sometimes take a very long time in the Galactic plane survey. RGM asked if this would cause problems for PSF fitting. DWE said that with the current version of the software, all the hard work had been done by the catalogue generation programme in performing the deblending and that the PSF fitting itself was quite quick. MJI said that we need to consider what lessons we have learnt from these reductions and factor in various improvements before the next batch of data arrives. He also noted that several aspects of the pipeline had improved since the first batch of data (April) had been processed and that the catalogues will also all need regenerating once the parameters have been finalised. This will eventually include PSF V1,2 information. DWE asked what happened to badly interleaved data. MJI said that the grouped frames are thrown away, but that the individual frames are kept. Similar questions were asked regarding frames where chip 2 has failed. The data is kept, but the data quality control will indicate that the frame requires reobserving. NAW asked about whether the CASU DQC will be fed back to JAC. This will be the case, but the JAC DQC should pick up most of the problems. EGS reported on the analysis he has been carrying out on the SV1 DXS data. The relative astrometry has an accuracy of 0.03" and the photometry 3% (for 13-17 mag). Comparisons have been carried out with other surveys. The DXS stack goes down to 20.6 mag in K (5 sigma limit in 2 hours). It was noted that the UDS data overlaps with some of this, but there is a problem with the 3x3 interleaving. EGS said that the report wasn't quite ready for general release but that he would distribute the web location to the group for comments. <<<< MJI said that he would be attending the UKIDSS survey heads meeting at Durham tomorrow. 6. VDFS update PSB reviewed the latest observing templates, one issue of note was that a single on/off command for the flat lamps had been provided, but after discussion will now be a 3-bit code to control the three sets. Some other minor discrepancies were passed on. MJI reiterated that STH needs to finish the ETC document. PSB noted that SMB has said that "what's important is the list of parameters associated with each template. It would be good to reach a general agreement on those parameters before submitting them to ESO". ESO will then release a VISTA-enabled version of P2PP (ESO Phase-2 Preparation tool) which can be used to generate OBs. The ESO DRS (Data-Reduction Specific) keyword proposal, originally due in the spring, has been acquired. PSB remarked on the extremely concise wording in the proposal. WJS reported on a number of VISTA hardware issues. A test set of detectors being cooled together with an autoguider CCD so that tests including pickup and cross-talk can be carried out on them. One of the three camera corrector lenses is still in the process of being polished. QEs at Z and Y of ~60-70% have been measured. Issues regarding inter-pixel capacitance have been raised at recent meetings. These cause an small coherence in noise properties between pixels such that the noise covariance matrix has a modest off-diagonal power (~20% for Hawaii II, unknown? for the Raytheon detectors for VISTA). This causes, for example, the photon noise to be underestimated and hence overestimates things like the gain by similar amounts. WJS noted that VISTA should be on the sky by the second half of next year. There was some concern that there may be pressure to cutback in the testing phase in order to save money. This was regarded as a false economy. 7. AstroGrid liaison update No news is ........ 8. AOB - ADASS 2005 The next ADASS will be in Madrid on 2-5 October. JRL has not quite <<<< finished writing the abstract yet. It is expected that JRL, PSB and EGS will attend. We were reminded to keep our .away files up-to-date so that we can work out, among other thngs, when to have the next meeting. <<<< Continuing Actions ------------------ MJI finish off report on the detector properties as they currently are JRL include section on processing strategy and illustrate with MR processing examples. Also finish EGS's SV report. EGS Deadline: a week before the SV meeting in Edinburgh (18 August). JRL finish off download side of AAO archive scripts (further email prodding or a telephone call required) WJS find out status of new version of the SADT Definition Document STH drive test version of the SADT available for both VISTA and VST MJI draw up a draft Q3 plan and circulate for comments ALL review Q3 plan by the next VDMT meeting (11 August) MR improve look of front page of WFCAM raw data archive JRL sort out scripts for access to the summit comments database and make available MR put some explanatory text in the processing progress web page STH finish the 3 remaining RIXs in the ETC document (by 5th August) PSB check the status of the various VISTA documents and ensure they are up-to-date WJS check that VISTA template definitions are sensible MJI raise calibration/science frame issue for VISTA at next VDMT PSB wait for response from JPE regarding the status of the SADT with respect to the RIXs New Actions ----------- MR include access to "observing" log index files in raw WFCAM archive JRL investigate the sky calibration frames to see if there is any correlation between Moon angle and scattered light EGS make stacking software available for specific requests EGS finish and distribute location of SV1 report ALL send back comments on SV1 report to EGS JRL finish writing ADASS abstract ALL keep .away file up-to-date