22nd August 2007 Adhoc meeting in Hoyle Committee Room Present: JRL, STH, EGS, PSB, MR, MJI, DWE Apologies: Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Data archives update - AAT, ING, WFCAM, UKIRT 4. Optical processing update 5. WFCAM news - raw data transfer status - WFCAM network transfers - processing status - reprocessing requests - photometric calibration update - operational lessons learnt 6. VISTA news - pipeline progress - cold camera test data from Paranal - VISTA QC and operations 7. ADASS 2007 - latest news 8. AOB - retiring cass36 - system management Minutes ------- 1. Actions from last meeting RGM have not completed the work on the classification aspect of cross-talk EGS weeding and classification generally - ongoing <<<< MJI MJI has still not put the OMP log comments into the processed directories - ongoing <<<< JRL are still working out a scheme for doing header updates for ING archive EGS - ongoing <<<< PSB has moved office to APM building and has ordered and installed another Gbit switch 2. Comments on WFAU minutes PSB reported briefly on his further investigations into data transfer rates to WFAU noting that WFAU have recognised the djedefre bottleneck. He also commented that a few WFCAM transfer sessions have taken place recently over UKLight at the usual 3MB/s, compared to realistic test WFCAM transfer rates (ie. a ~50 GByte whole night of real processed WFCAM 06B data) of ~30MB/s using un-encrypted hpn-ssh to seaforth connected over UKLight. STH commented on the release schedule for DR3 and wanted to clarify when T0 was likely to be and were the units in the minutes really weeks since they implied DR3 would be 14.5 weeks after T0. MJI thought they were unlikely to be anything other than weeks but noted that some of them happened in parallel so the time after T0 was unclear. He also noted that since all of the 06B data had finally been processed, checked and flagged, T0 should be imminent. 3. Data archives update Most of the off-line primary archive tapes have now been moved to their temporary location viz the lockable store cupboard in the Observatory meeting room. MJI noted that the "cleared" shelves in the APM store room could be used for storing personal backup tapes and so on. ING: Data continues to flow seamlessly from the INT and WHT via automated network transfers and a night's observing is invariably completely transferred to Cambridge before the observer has made it back down to bed. AAO: New nights of data are transferred over the Internet as they are made available. The new access interface will be publicised in the next AAO newsletter. Small bug fixes are still being made, but the system seems to be working well on the whole. UKIRT: This is up-to-date. WFCAM: All 06B data that are available at JAC have now been received and ingested into the raw data on-line archive. 07A data was received long ago and 07B data has not yet started to flow. MR noted that roughly 2 weeks of 07B data (allowing for stoppages due to Flossie) had been taken and he would inquire when the first tapes would be shipped. <<<< Transfer of remaining UKIDSS and calibration raw data for 06B to ESO is ongoing. 4. Optical processing update The IPHAS processing and DB ingestion for June and July has been completed amid much excitement due to jammily capturing pre-Nova r,i,Ha images of Nova Vul 2007. EGS has developed and released a further trial component (alpha stage) for simplifying browsing of the Astrogrid-enabled IPHAS Early Data Release. 5. WFCAM news All of the available WFCAM raw data for 06B has arrived. PSB has had an email exchange with Andy Adamson regarding the possibility of transmitting raw WFCAM data to Cambridge via the Internet. An outline plan was sent to JACH and we are waiting for Brad Cavanagh to reply. (It is hoped to reduce manpower overheads, increase data integrity and improve lead time to science delivery.) The final few nights of 06B data have now been processed and as of today all of the data for 06B that is processable has been processed, checked and flagged as ready to go. There are a couple of problem nights, in particular 20061116 and 20061118. The former has most of the data missing but has been processed as well as possible. The latter has no #3 data for the entire night and has been flagged as un-processed after a brief email exchange to verify this with WFAU. This completes the processing of 06B. The quality of the October/November data up to the 20th was in general quite poor due to: bad weather, lack of enough good calibration frames (especially darks), many missing frames (caused mainly by camera problems), and a lot of interrupted incomplete group MSBs. MJI and MR summarised the state of play with reprocessing requests: The problem GPS data for 06A has all been reprocessed and Phil Lucas has checked it. Most of the requests for reprocessing concerned fixing the known feature in the extant (at that time) cross-talk correcting algorithm that made a pigs ear in regions of highly varying nebulosity. A few were related to poor sky sbutraction but as these were invariably on poor nights interrupted by bad weather these were harder to deal with (see previous minutes for more details). The UDS saga continues. JRL will test a special purpose sky background modelling algorithm involving iterative object masking on some of the UDS data that has been requested for reprocessing (as above, see previous minutes for more details). STH reported that he will shortly (end of the week) circulate a final analysis of the WFCAM Z and Y-band zeropoint offsets to the calibration WG for comment, prior to implementing the adjustment to the processed catalogues. MJI noted that to avoid confusion he would not apply the upcoming new offset corrections to the processed WFCAM data until the remainder of 06B has been transferred. He also commented that since this is currently planned to be purely a zero-point offset for these two bands only it should be trivial to implement at CASU and WFAU without the need for any transfers. MJI kicked off a general discussion about lessons learnt from the operational side of running the WFCAM pipeline for several semesters. PSB and JRL commented they knew it all already and had already built in the necessary infrastructure to the VIRCAM pipeline - we shall see. There was discussion at length about software updates, tests and the ability to trace changes made in the pipeline infrastructure. In spite of the previous optimistic statement, JRL agreed to go away and think about this some more over the coming month or so. MR contacted the Durham group (Jim Geach) about their progress in processing raw WFCAM data and obtained some example processed data. Now that 06B processing is finished he will undertake a comparison. <<<< 6. VISTA news JRL commented that he had received some requests from ESO for updates to the pipeline. One of these comes about from the change to CFITSIO and its impact on reading binary tables. The others were for minor (but irritating) name changes to various parameters etc.... PSB reported around 2,300 frames were obtained from the July Paranal engineering run of VIRCAM, with apparently only two being lost semi-permanently. Because of some technical difficulties with the camera, the data needs reformatting to be able to be processed (semi)automatically. This is in progress. JRL and PSB are currently running tests on this camera data. This has already turned up a few bugs, albeit minor ones. A report on the data will appear in due course. MJI suggested that a further independent examination of the VIRCAM test data was probably worthwhile and volunteered EGS to look at it as well since MJI and MR were busy. <<<< 7. ADASS 2007 The final programme is currently being constructed and will be sent to the printers next week. A floor plan and final decisions regarding catering will also need to be done in the next week or so. The conference looks as though it will be well attended with probably a little over 300 people. 8. AOB MJI and PSB noted that cass36 was now (definitely honest) going to be retired, albeit in gradual installments. All thanked JMI for his expert system management over the past few years. PSB and MJI were tasked with acquiring a system brain dump from him before he leaves to take up a postdoc position at Harvard. <<<< All congratulated Simon on his recent 40th birthday. Continuing Actions ------------------ RGM report back on classification aspect of cross-talk weeding EGS MJI MJI place OMP log comments into processed directories JRL work out scheme for doing header updates for ING archive EGS New Actions ----------- MR find out when the first 07B tapes will be shipped MR undertake comparison of Jim Geach processed data with pipeline output EGS help with assessment of VIRCAM lab data MJI find out how little they know about CASU system management PSB