Monday 18th May 2009 APM Meeting tables 11:30-13:00 Present: MJI, JRL, STH, SC, RWA Apologies: NAW, MR, EGS, RGM, DWE Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, Gemini, VIMOS 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- 1. Actions from the last meeting STH PLONE pages. Sky subtraction and dribbling text updates have been added RWA (it later transpired that the sky subtraction text should also be on the WFCAM as well as the VISTA technical pages.) RWA will continue checking the technical pages - instances of poor alignment noted. <<<< STH HAWK-I colour image of Carina. STH has put this on a private web page with password and demo'd zoomifying at the meeting. JRL still has to write some text to go with it. <<<< MJI says (re)photometry for HAWK-I still to be done <<<< MJI Stage 1 release software - still ongoing <<<< EGS MJI updates to technical paper started, but not yet finished <<<< RGM liaised with JRL about HAWKI - data processed JRL has sorted out the VISTA docs and pdfs so that this is essentially complete and they are all now on the VDFS internal web pages. MJI write utility to convert binary tables with all corrections applied - in the queue but not high priority <<<< JRL has sent the latest catalogue software to WFAU JRL implemented keyword in sky FITS files and back copied it to all 09A sky files EGS has not yet sent off example code to UVEX team <<<< NAW polite but firm e-mail to VIZIER about IPHAS - ongoing <<<< MR sorted out WFCAM transfer script with result that MJI now gets two e-mails per day from MR rather than PSB STH inquired about linearity sequence status and noted nice new screen but no lamps yet MJI wrote and sent the two reports to JPE for distribution JRL MJI has put the minutes he has of the IOT telecons on the internal web pages, though he noted that the minutes of the previous April meeting have not appeared yet STH is liaising with ASTEP folk and has started to construct a webpage <<<< MR status of comparison of FITS file header problems on 20090312 with original sdf files unknown - ongoing <<<< MJI compiled and sent off report to AJA for the upcoming UKIRT Board meeting and put a copy of it and the UKIDSS submission on the internal web page JRL has been in communication with JAC about cfitsio version in use there JRL VISTA processing status webpage. A lot of action on this and it is now, MJI thanks to MR, doing live updates. JRL has also has made access to reduced data possible by our usual password-protected web page access. MJI is steadily verifying, correcting astrometry and producing summary QC plots for the live web page. 1st pass photometric ce's and ZPs have been generated based on STH touchstone field observations. <<<< MJI sorting out the VISTA i/o disks is on hold until the GAIA team finish assorted i/o tests on the new RAID system. <<<< 2. Comments on WFAU minutes MJI noted that as a result of the recent computer attack on the IoA cluster, all private authorized keys on the APM cluster were disabled as a security measure. It wasn't clear at the time who was using this method but it turned out to be the ING incoming transfers and the WFAU outgoing transfers. The relevant keys have now been regenerated. There have been several comments about processing and reprocessing status of various things - see the WFCAM section later for the current state-of-play. 3. Recent and upcoming meetings There will be a UKIRT Board Meeting in Nottingham on Thursday, and SJW, NCH and MJI will attend for the pm session. EGS later reported via email that he gave a CASU presentation in the VO and Distributed Computing session at NAM 2009. He noted that it looks like Europe and the US are taking the VO seriously. VISTA IOT telecon on April 28. JRL and MJI attended from Cambridge and STH from Mauna Kea (albeit briefly due to inclement weather). JRL and MJI submitted reports on sky subtraction strategies and science verification results and they can be found under the VISTA commissioning internal web pages. The next telecon is due in about a week. There is a definite need for a sky offset template (HAWKI uses 4 different flavours, but 1 would do for VISTA) as noted during the previous telecon. After a brief discussion on possible practical ways to implement this it was felt that this should be taken offline and the constraints discussed with JPE and Steven Beard before proceeding further. STH noted that as he was imminently about to fly out to Paranal he would discuss this further with JPE when he arrives. MJI noted that the PLATO Science consortium will meet in Belfast on June 11. STH can't go but NAW will attend and further impart some wisdom. JRL commented that ADASS 19 is now open for registration (Sapporo, Japan) and he thinks we ought to contribute something on VISTA processing. MJI agreed but pointed out that the travel budget is currently zero and still awaiting a delivery from STFC. At the JENAM meeting Tim de Zeeuw said that first light for VST would be this September - watch this space, but don't hold your breath for the first survey data. 4. Data archives AAT - Jim says that the AAO now write to DVD in real time. He has written a script to download the data as each DVD is completed and we are now running ~1 night behind. MJI noted it might not be a bad idea to use a semaphore system of some sort rather than dead reckoning to ensure synchronicity. <<<< The RAID array that this is being stored on seems to be having a hard time of it recently and has had two different disk drive units going awol within the space of a week. No obvious reason other than stochasticity but at the time of writing another (different again) channel is going awol - d'er! ING - data keeps on rolling in and being verified, ingested and backed up to tape. WFCAM - MR has more fully automated the MEFing and verification step, which together with the automatic data transfer stage from JAC to CASU has further minimised overheads for this aspect (good job too!), but still needs manual tape archiving further downstream. UKIRT - remaining few DLTs and LTOs still to read VISTA - MJI noted a report in the latest ESO news letter that from mid-2009 bulk ESO archive users need to be ready to receive Blu-Ray disks (sort of souped up DVD) and it warns users to have suitable hardware in place. MJI asked JRL to double check that this is not going to affect us and that the disk bricks will still continue to be the VISTA transport medium for the near future. <<<< 5. Optical/NIR processing HAWKI - as noted previously, JRL has processed the latest HawkI data for RGM and MJI still hasn't finished off regenerating the Carina catalogues. Yet another excursion for the nebulosity filter. INT WFC - MJI had a pleasant observing run on the INT WFC towards the end of April, generated and analysed some new non-linearity observations and finished the processing of data from his observing run before he got back. MegaCam - nothing to report Subaru - EGS says he has finished processing deep optical data from Subaru, pipeline runs smoothly and produced beautiful stacks. Gemini - MJI processed some SLUGS imaging g,r,i data for an IoA-led lensing consortium VIMOS - a pile of venerable P74 and P75 deep VIMOS imaging data on nearby galaxies has also been processed to final stack stage. All that remains is to trim the crud, remove the assorted reflections and generate some catalogues. 6. WFCAM update. 09A processing is going smoothly after a slight pause to generate a full set of May flats before starting the May processing. MJI is doing the honours while MR takes a well-earned break observing with WFCAM, or maybe the break is the holiday bit after. As usual when at full steam the pipeline processing to completion runs about 5-7 days behind the date of observation (but recall the observation "date" is UT-based and it takes ~24 hours from observing finishing for the data to be transferred, verified and ingested here). At least one logjam could finally be cleared now that sufficient 09A data has been accrued to compute the photometry corrections. The final checks and photometry udpates were made to January-April 09A data over the w/e and this has now been flagged as ready for transfer to WFAU. A disk hardware failure on a RAID unit on one of the processing engines required a disk drive to be replaced. The bad news is that this flavour is now out of warranty, the good news is that we have around 8 spares from a now defunct older RAID array. Reprocessing is currently stymied somewhat by insufficient hours in the week but moaning aside the current status is: the UDS K-band reprocessed data awaits the final thumbs up from the UDS team but is looking good. MJI was asked to help make a quick assessment of the depth (expected -v- reality) and predicted from purely photon noise limitations due to sky, a 5-sigma depth of K=22.3 within a 2 arcsec diameter aperture from the no. of components in the latest stack, or ~22.1 for an average point source - i.e. accounting for aperture loss. His analysis of the random noise component (allowing for interpolation correlated noise) of a 10kx10k chunk of the proposed deep stack suggested K=22.2, in reasonable agreement. Larger scale correlations present in the deep stack will conspire to lower this limit by ~0.1 mags but this is now fairly close to what is being measured by the UDS folk. MJI will double check with Omar at the Board meeting before making this available for transfer and also discuss with NCH the simplest approach to this. MJI finally started the GPS nebulosity reprocessing after writing a script to do all the hard work. The filtering took a few days and the catalogue regeneration several more. It now needs assorted tests running in conjunction with GPS folk. Lack of available effort has meant that the Petrosian etc. reprocessing/ updates for 05A-08A still haven't been started and neither has supplying WFAU with the relevant script/details. MJI noted that there is an upcoming PPRP meeting in Glasgow at which, among other things, a proposal to build a planet-finder spectrograph for UKIRT will be discussed. He noted that if successful we could be involved in the archiving aspects. There has also been a flurry of e-mails about an old hot potato, flushing the detectors after changing filters. In practice this amounts to a single "quick look" recipe exposure of similar parameters to the upcoming science observations after each filter change. For some projects this adds negligible overheads (~1%) for others it can be up to ~5% or more. The need for this is going to be reassessed. It mainly (solely?) seems to affect detector#1 and was first spotted in SV data taken in April 2005 by SJW. 7. VISTA update CASU have received complementary remarks from Thomas Szeifert on the quality of the output from the summit pipeline. JRL notes he has made a few more bug fixes (Shirley not) to VIRCAM software and is making good progress on the UK science pipeline. No new useful science data has yet been received but we anticpate once the detectors are running at the correct operating temperature and the camera is back on the telescope that characterisation will begin in earnest once more. STH has flown out to Paranal to help JPE and WJS with this process. MJI asked about the progress with the tiling software, tests with MONTAGE, and in particular what the contents of the tile FITS headers will be, i.e. what gets inherited, what doesn't, what extra keywords etc... JRL said he would produce an example tile FITS headers that could be used as the basis for discussion with interested UK parties. <<<< As noted earlier the VISTA processing status webpages are now live after a lot of recent work. They are accessible remotely but deliberately are hidden from casual view. All of the useful processed VISTA data up to the end of January is being checked astrometrically and photometrically and the usual WFCAM style QC information is being generated. The first tranche of this is available through the web page and the first pass colour equations and default zero-points have been produced. New raw data is picked up and logged on these pages as it comes in after being copied, verified and ingested. MJI commented that JPE was planning to spend his upcoming sabbatical at the IoA and given the current STFC funding crisis we can put him to good work helping to run the VISTA pipeline. 8. AOB The TV/wall mounted monitor for the CASU meeting area has been in abeyance for a while and as SC raised this topic MJI asked SC to investigate the possibility of a display similar to that on the wall outside the Sackler lecture theatre -v- a large format LCD TV (which was the original plan). <<<< Continuing actions ------------------ RWA continue checking through the technical pages looking for glitches et al. JRL produce web page blurb for HawkI MJI finish off HAWKI reprocessed data by nebulosity filtering and recataloguing MJI finish off stage-I software release pages EGS MJI finish off technical paper updates MJI write utility to convert binary tables with all corrections EGS send off example code to UVEX team NAW polite but firm e-mail to VIZIER about IPHAS reqired STH put ASTEP info on webpage MR check with JAC to see if SDF files have same detector#4 header problem MJI continue producing summaries a la WFCAM of reduced VISTA data MJI sort out VISTA i/o disks New actions ----------- JRL set up semaphore system for AAO transfers JRL check any impact of ESO archive decision to go to Blu-Ray disks JRL produce draft FITS headers for tile products SC investigate TV/monitor similar to that in the Hoyle for CASU meeting area.