12th June 2007 Observatory Meeting Room 10:30 - 13:00 Present: DWE, STH, JRL, PSB, MR, MJI, RGM, RWA Apologies: NAW, EGS, RG Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Rolling grant submission 4. News from meetings - VDMTs, VDUC, UKIRT board, UKIDSS 5. Data archives update - AAT, ING, WFCAM, UKIRT 6. Optical processing update- VST delays - IPHAS release 7. WFCAM news - raw data transfer status - processing status - photometric calibration update 8. VISTA news - pipeline progress - summit processing hardware - cold camera tests lab and on telescope - draft milestones 9. ADASS 2007 - latest 10. AOB - retiring cass36, cass123 - attempted break in - plates and tapes in SPO Minutes ------- 1. Actions from last meeting RGM have not completed the work on the classification aspect of cross-talk EGS weeding and classification generally - ongoing <<<< MJI MR has sorted out the credit card payments for ADASS 2007. Ian Bell has JRL access to a credit card machine and the information for this (entered manually) will come from the web form. MJI raised the issue of field rotation flat field tests and VISTA integration at a VDMT meeting (see later) MJI raised the issue of updating the VDFS UK User Requirements Document at a VDMT meeting. This document now has been updated. NAW emailed Tim Naylor (probably) to obtain agenda for VDUC meeting and minutes from previous meeting. These were then distributed. MR contacted the Durham group (Jim Geach) about their progress in processing raw WFCAM data. They are happy with their reduction. MR was asked to obtain some example images from them for comparison. <<<< MJI asked about the possibility of obtaining more VISTA test data at the next VDMT meeting (see later) NAW have discussed the possibility of fast links to ESAC (for Gaia). PSB This item will be continued offline. EGS has not investigated photometric redshifts in DXS - ongoing <<<< PSB has sent his FTP test results to Brad (CC'd to Andy Adamson) and this has shown that the transfer of (possibly) all WFCAM data is feasible. PSB was asked to prod JAC to see if anything was going to come out of this eg. transferring the missing NDR data. <<<< MR inquired about the causes of the intermittent nature of WFCAM transfers to ESO. Jens Knudstrup replied and said that the software had been checked and was OK. The most significant reason is that there is disk downtime caused by having to take various machines down to change disks. The nominal bandwidth for transfers is 2.0 Mbytes/s, but what is achieved is 0.4 Mbytes/s due partly to the ESO-end ingestion process. JRL has been in touch with Kognitio regarding further tests involving their data base software. Although a test doing a cross federation of IPHAS and 2MASS data has not been carried out, a simple calibration test on IPHAS data has been very successful. While the test took 20 days using a single-thread Sybase system, the Kognitio system took 8 minutes! (to be fair they had many more processors, but it does show that this is a scalable task). Kognitio have agreed to give us a demonstration copy of their software and JRL and PSB will be carrying out various tests on it. <<<< EGS have sorted out the use of the spare 750 Gb disks as backups for the MJI IPHAS data base. MJI has double checked that there are two extra spares. MJI has still not put the OMP log comments into the processed directories - ongoing <<<< MJI have looked into the light leaks problems in dark frames. A consequence MR of this is that the generated darks for each night processed have to be checked manually. These light leaks have been detected as far back as 06A. STH has updated his secret webpage on the recent photometric calibration developments. MJI emailed NCH to remind him about the photometric corrections needed to account for the astrometric distortion MJI did not have to raise the issue of what is meant by exposure time value contained within VISTA FITS headers at the VDMT meeting since it was picked up directly from the minutes. After a flurry of emails an SPR has now been issued. ALL provided inputs for the rolling grant application which was submitted on Friday. 2. Comments on WFAU minutes It was noted that UKLight is working, but it is thought to be a bit slow (3 Mbytes/s). This is fine for the current workload, but may be a problem for when VISTA and WFCAM data need to be transfered. STH drew attention to the problems GPS is causing at the archive end of operations. There was a short discussion about the DR3 release. No date can really be given for this until CASU receives the missing NDR data. Once that has arrived, it should be processed within a month. 3. Rolling grant submission MJI reported on the recently submitted rolling grant application. 4. News from meetings There have been 2 VDMT meetings (27 April & 8 June) since the last CASU meeting. Various issues arising from our actions were raised. The main gripe was the release of the pipeline software version 0.6 to ESO without going through the VISTA project office. In mitigation 0.6 was a minor upgrade to fix "tickets" arising from the major release 0.5. PSB and MJI attended the VDUC meeting in Edinburgh 10 May. After this meeting a new UK URD was produced (much cheering). There has also been a lot of useful feedback from the PIs and more to follow after the release of the updated URD to them for comment. STH attended the UKIDSS Survey Heads meeting in Oxford 9 May. Topics discussed: date for DR3; new scheduling log for surveys (allocations of time); award of time to 3 new campaign surveys (including STH's Transit Survey). STH noted that the world release for EDR would be 10 August and for DR1 21 January next year. Omar Almaini commented at the meeting that some of the UDS data for 06B needs to be reprocessed. MJI said that he had an email about this yesterday and will investigate. The next Survey Heads meeting is supposed to be at the end of June, and will be in Cambridge. Following the Oxford meeting (10 May), STH attended a calibration meeting with Steve Warren and Paul Hewett in which they spent half a day going through the calibration paper and progress. Both seemed happy with it and the next meeting is scheduled for the first week of July. MJI reported on the UKIRT board meeting he attended on 17 May in Durham. All the reports to the UKIRT board are available on our web site http://casu.ast.cam.ac.uk/documents/other-meetings/ukirt/ and the presentation can be found at http://www.ast.cam.ac.uk/~wfcam/talks/ukirt_may2007.ppt The board seemed happy with the data processing and were particularly taken by the diagram showing the growth of data volumes compared to other surveys. 5. Data archives update ING: RWA reported that 300 VMS backup tapes were taken to La Palma in order to extract the data from them. 60% of the tapes were processed resulting in 7000 files in SDF format that need to be converted to FITS. The transfer was quite slow since data also has to be copied across to a different machine afterwards. Most of the data is from WHT Taurus, FOS or direct imaging. The tapes have been left out there to be processed at a future date. JRL said that no progress had been made regarding the header updates for the ING archive. He and EGS will work out a scheme which RWA can use. <<<< The store room in the SPO containing our tapes and plates is being gradually emptied. The backup tapes will be stored in the annex between the Obs building and old RGO building. Many other tapes (and CDs) will be thrown away since they overlap with other media. DWE expressed a concern that the annex will get very hot in the summer and that this would not be good for tapes. RWA said that he would ask Peter Weeks if blinds can be fitted. <<<< AAT: JRL reported that the AAT archive is now working including the synchronization between their data base with the observing logs and ours. http://archive.ast.cam.ac.uk/aat/ This archive contains data from 1990 to the present. There was a discussion on how to advertize this. There already is a news item on the Plone site, but STH felt that more was needed. The consensus was that the advertising should come from the AAO side and JRL said that he would email Stuart Ryder to expedite this. <<<< UKIRT: This is up-to-date. WFCAM: Tapes have been received up to 21 May ie. we have up to all of 07A and 06B except for the NDR files from the end of October to 20th December. The copying of UKIDSS and calibration files over to ESO for the raw archive is complete up to 23 April. 6. Optical processing update MJI reported on the VST delays. It is unlikely that any public surveys will start until well into 2009. A large part of the band-merged IPHAS products have been released within the IPHAS consortium for limited testing using AstroGrid software. RG is currently working on moving the remaining IPHAS web pages at ING over to Cambridge. CASU are also still carrying out the extinction calculation derived from the Carlsberg Meridian Telescope data. DWE said that there is still a payment due for this work which needs chasing up. MJI said that he would ask Paul Aslin about this. <<<< 7. WFCAM news MR said that processing was complete up to 20 April except for a small chunk of January data that is still being processed. MJI gave an update on the photometric calibration. Half of the illumination correction required is at the detector level. This probably arises from pedestal offsets within the flat fields since the offset patterns are stable over a flat field period (ie. roughly monthly). It is simple to remove this part of the systematic effects using the individual detector FITS header keywords. The remaining systematic patterns appear unrelated to the flat fields and seem fairly stable over semester-length periods suggesting that a correction table could be used to further reduce their impact. CASU have been asked about how asteroids are dealt with during the data processing. The short answer is that they are ignored since this was always felt to be an end-user/archive issue. A further 20 Tbytes of WFCAM data storage have been purchased (apm24_c/d) and installed and a bit of data shuffling has started populating (and testing) them. 8. VISTA news Maintenance issue 0.6 of the pipeline software has been sent to ESO and deals with the gripes about the 0.5 version. In general, things are looking good. There was a brief discussion of a draft milestones document. The cold lab tests of the camera are being carried out at Paranal and it is planned that camera will go on to the telescope for EMC tests in July. CASU have been asked to help investigate the results. Related to this is the issue of how to get access to the data there. PSB has requested (via JMS) access to an account on the VIRCAM system. There was a brief discussion about what further tests could be done. PSB is already working on this. One extra possibility is to attempt flat field rotation tests even though the illumination will not be uniform it should still be possible to assess possible problems. PSB reported that the summit processing hardware may only have 1 or 2 processors, rather than the 16 processor blade machine originally foreseen. If this goes ahead it will limit the amount of real-time DQC that can be carried out. MJI said that the revised SMPs (even RGM's) have been submitted. Documentation was then discussed, in particular the Calibration Plan. MJI will email JPE regarding version 1.5pre1 of this and will request the pending v1.4 be released in the meantime. <<<< 9. ADASS 2007 JRL said that there was no real news. The hall cost has gone up by 2%, but this was anticipated. A decision was made that there will be a nominal charge for the tutorials. 10. AOB cass36 and cass123 will be the next machines to be retired. cass123 is just coming off maintenance and will be naturally retired. cass36 should only be serving access to disks. A similar piecewise removal operation will be carried out as for cass38. There was an attempted break in at the back of the APM building last week. Steel plates will be placed over the damaged woodwork. MJI noted that we should ensure the blinds are closed in the evening and at weekends and also make sure that laptops or other visible goodies are not left lying around. STH raised the issue of office space in July. He has many visitors and some of his visitors have invited their own visitors (d'er!) It is unclear where these folk will go and STH will liaise with MJI offline to solve this. PSB pointed out that there are next to no spare slots in our network switch gear. He was therefore asked to order another Gbit switch for the APM building to solve the problem. <<<< Continuing Actions ------------------ RGM report back on classification aspect of cross-talk weeding EGS MJI EGS investigate photometric redshifts in DXS MJI place OMP log comments into processed directories New Actions ----------- MR obtain some example images from Durham group (Jim Geach) in order to do a comparison PSB contact JAC regarding FTP transfers eg. transferring missing NDR data JRL carry out tests with demonstration copy of Kognitio software PSB JRL work out scheme for doing header updates for ING archive EGS RWA ask Peter Weeks if blinds can be fitted to interconnecting annex JRL email Stuart Ryder about advertising the AAO archive MJI ask Paul Aslin about the Carlsberg extinction payments PSB email JPE regarding version 1.5pre1 of Calibration Plan PSB order another Gbit switch for APM building