28th June 2006 Hoyle committee room 13.00 - 14:10 Present: DWE, MJI, JRL, PSB, MR, EGS Apologies: WJS, RGM, STH Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report on VISAS meeting 4. Data archives update 5. Optical processing update 6. WFCAM news 7. VDFS news 8. AOB Minutes ------- 1. Actions from last meeting MJI has made no further progress investigating the merging of classification statistics - ongoing <<<< WJS has not contacted the VPO re: ETC-related digitized data. This and the related action have been deprecated after an email from JPE. PSB commented that JPE has acquired some mirror reflectivity data (but has not passed this on to us yet). JRL has not implemented the fix to the problem with the UKIRT archive (missing header information problem) - ongoing <<<< EGS has not updated the INT WFC web pages with the latest flat-field and fringe frames - ongoing <<<< STH it is not known if the data from the 2MASS Touchstone fields has been acquired (STH not present). This is needed for further linearity tests - ongoing <<<< DWE have updated the hardware whereabouts page (HTML and Plone) MJI PSB had a short meeting to finalise updates to DRLD JRL MJI DWE are still finalising content of Plone site - ongoing <<<< EGS STH MR 2. Comments on WFAU minutes The previous "Rant of the Week" concerning the UDS processing was noted. CASU agrees and extends its sympathies. MJI noted that the latest minutes stated that the agreed single thread network transfer test would be carried out after DR1 has been released. 3. Report on VISTA PSP meeting JRL reported on the VISTA PSP meeting he attended in Edinburgh on 20 June. The main activity at the meeting was horse-trading between the original proposals, which was very successful. The original set of proposals shows a 3 to 4 times oversubscription in the available observing time. By merging some of these proposals, the amount of oversubscription was reduced. From the 10 surviving proposals at the start of the meeting, there now remain 6 (Magellanic Clouds, Galactic Bulge, VIDEO, VIKING, VHS, a UDS). What JRL found worrying was the way many of these projects were intending to take data. There is a need for us to give clear advice on observing recipes particularly those proposals that do not want to dither. Many are concerned at the overheads reported by the ETC, believing they will maximise their science return by not dithering. PSB pointed out that it was very important to dither given the bad pixel distributions and notes that astronomers may not be aware that the acceptance level for the proportion of bad pixels per chip is 5%. JRL said that we have been asked by the PSP (via Marc Casali) to produce a crib sheet of observing strategies. There is clearly a need to re-examine the overheads used in the ETC due to the recent changes with respect to parallelization eg. writing data to disk while filter changing or dither offsetting. PSB reported on a phone call where JPE said that he would write a specification for the overhead algorithm. This will be forwarded to ESO for the official VISTA ETC and to CASU for possible inclusion in the test ETC. However, MJI also noted that in his experience overheads were rarely overestimated and erring the other way was even worse. JRL also reported that there seemed to be a distinct lack of manpower considerations for data reductions beyond that provided by the CASU pipeline in these proposals. ESO DMD will undoubtedly pick up on this. The merged proposals will now be rewritten and resubmitted to the selection panel including revised data management plans. 4. Data archives update - AAT, ING, WFCAM, UKIRT AAT - no change ING - EGS said that the data for 1996 had now been refactored into date subdirs and duplicate entries flagged. The period 1996 to 2006 is now almost complete excepting some INT data from the latter half of 2004, around the transition between DVD and Internet transfer of data from La Palma. MJI and EGS will arrange for RWA to fill in this gap and stress test some more DVD readers. <<<< WFCAM - MR said that tapes have been received up to 15 June. All the data up to the 7th of June have been MEF'd and are available through the raw data archive. He is currently dealing with the last set of tapes, 8th-15th of June. Transfer to ESO of 06A UKIDSS and calibration data has commenced. JRL said that he had a nephew coming for a couple of weeks for work experience as part of school course requirements and that helping with reading tapes and MEF'ing files might be a suitable task. We will see. UKIRT - no new data tapes. 5. Optical processing update EGS said that he had been reducing some INT data for Dan Smith and Martin Jarvis. 6. WFCAM news (Non)linearity saga - this one could run for a long time. JRL says that his analysis of the photometric test set (same field but exposure times ranging from 1s to 60s) reveals no clear evidence for non-linearity until close to saturation. MJI asked to be pointed to the dome sequences taken on the same day so he can carry out further checks. <<<< MR reported on the progress of 06A reductions. MR reminded us that most of the May data has assorted randomly missing chips and also a bunch of files with null values for detector #2 (as reported in previous minutes). After several modifications the pipeline now deals with these more robustly. A small number of files have also been found with incorrect offsets in the header. MR was asked to email Paul Hirst (CC Andy Adamson) concerning this and any other problems he has found. <<<< Since 2 June, all chips are present in the files, there are still some null fulls for chip#2. This is still about 1% of files ! MR noted that some MSB observations have been split in an odd way and this has caused problems due to the interesting way they have been deconstructed and then partially started again and continued. These MSBs are effectively unreduceable without much header fiddling. MR said that these MSBs were all PI data. MR also said that he and MJI have been re-reducing some 05B data for UDS. It was noted that a recent paper on the UDS (astro-ph/0606386) claims that our pipeline components are not optimal for deep stacking and cataloguing. MR and EGS said that they are looking into this independently <<<< MJI said that the additional air conditioning has been installed successfully and has been running for the last 2 weeks. This provides an extra 7kW of cooling capacity and should keep the CPUs and disks happy in the soon-to-be ex-store room. Together with the previously installed kit we now have the wherewithall to create an artificial indoor ski slope or ice rink. Two existing computers/RAID arrays have now been rack mounted (rather than desktop mounted), cass03 has been finally retired, MJI finally got his new desktop machine after months of waiting, and much old (solaris) disk rationalisation has occurred. It is intended that two other computers (apm18 and apm23) will be moved from their current racks into the new racks in the store room to reduce the ambient noise level in the main APM area. There are no immediate plans, or requirements, to purchase any new hardware. 7. VDFS news PSB reported that he had a productive visit to RAL and that he has returned with 270 files - all with full headers. Currently, the RAL team are in the middle of intensive spot projection tests of VIRCAM. The data PSB obtained included linearity sequences, noise/gain runs, darks and 1 persistence run. It is unlikely that we will get any more data before commissioning on Paranal begins so we can deconstruct this data in earnest and see what we can learn about VIRCAM. JRL reported that the ESO end-to-end test had been very successful. We supplied simulated data to test out our supplied V0.3 recipe modules. ESO also carried out some archive ingestion tests using "real" VIRCAM data. There were a few minor header problems reported, but only 1 (out of 6) was of CASU origin. PSB said that it would be useful to configure the VIRCAM data-acquistion system at RAL to be close to the production system, such that most of the real overheads could be determined empirically to feed into the ETC. Such tests could be carried out with a warm, but complete, camera. 8. AOB None - record finish time - the absence of RGM was purely a coincidence. Continuing Actions ------------------ MJI finish investigating merged classification statistics JRL implement fix to problem with UKIRT archive (missing header information problem) EGS update INT WFC web pages with latest flat-field and fringe frames STH acquire data from the 2MASS Touchstone fields for linearity tests DWE finalize content of Plone site by end of month (which one ?) EGS STH MR New Actions ----------- MJI arrange with RWA to fill in the 2004 gap for the INT in the ING archive EGS JRL email MJI on whereabouts of dome linearity sequences MR email Paul Hirst (CC Andy Adamson) concerning incorrect offsets in file headers (and any other problems) EGS investigate UDS claims regarding pipeline components not being optimal MR