17th May 2005 Hoyle committee room 10.00 - 13:30 Present: MJI, RGM, PSB, DWE, EGS, STH, WJS, JRL Apologies: MR Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Global calibration of UKIDSS data 4. VDMT report 5. Data archives update 6. Optical processing update - VST surveys news 7. WFCAM telecon and processing update 8. Post-FDR VDFS meeting at ESO 9. Plone for documentation 10. AstroGrid liaison update 11. AOB - GAIA news Minutes ------- 1. Actions from last meeting JRL has still to include access to "observing" log index files in the raw WFCAM archive - ongoing <<<< JRL arranged next JAC telecon (see item 7) MJI raised issue of flagging observations of standard stars and other calibration frames at JAC telecon (see item 7) MJI raised issue of what is needed for the generation of master flats at JAC telecon (see item 7) MJI are in the process of writing a new report on the detector properties JRL as they currently are - ongoing <<<< EGS MR trained STH in the use of the pipeline MJI contacted Judith Moss re. getting a whiteboard MJI has not had a chance to produce a M51 colour picture - ongoing <<<< EGS has generated and distributed usernames and passwords for the Plone website ALL the discussion regarding Plone was postploned until item 9 EGS generated an option for hiding the side bars in the Plone-generated web pages STH had a meet with Guy Rixon and Kona Andrews to configure cass123 (see item 10) STH approached the system managers with a view to placing cass123 in dungeon, which has happened. Further discussion was deferred to item 10. 2. Comments on WFAU minutes It was noted that the WFAU minutes were getting shorter as ours were getting longer. This led to the postulation of a law of conservation of length of minutes within a project. MJI noted that there will be a UKIRT Board Meeting in Cambridge on 26/27 May at which MJI and NCH will give presentations. MJI said that he would find out who will be attending this meeting. <<<< 3. Global calibration of UKIDSS data EGS reported on a WFAU document on the global calibration of UKIDSS data. The approach was to gather all the information available (observation of standards, overlaps, etc.) and do a big non-linear solution all at once. For this they apparently need PSF magnitudes and illumination maps. There then followed a discussion on extinction and the best observing strategy for determining extinction. Views were also given on the merits of global versus non-global solution methods. It was decided that there was a need to respond to this document within a timescale of a couple of weeks. MJI said that he would place this document on our internal web page. <<<< EGS will collate the comments (aiming for about a page in size). <<<< There was a short discussion on the use of 2MASS and photometric standard fields for calibrating WFCAM frames. The effectiveness of calibration methods should be reviewed after the first period of WFCAM science observations finishes in June. 4. VDMT report The last VDMT meeting was on May 5th and a number of issues were raised and discussed: MJI reminded us that the VDMT UK Review will be held on 16 September and that we need to start assessing the amount of work required for the various extras needed. The data rate from VISTA has now been agreed and it is between a factor of 1-2 more than WFCAM. This is mainly due to the shorter than expected exposures of WFCAM and the longer exposures of VISTA. This suggests that issues of scalability of the pipeline are irrelevant. The UK design documents have been frozen (cryogenically), but are still awaiting signatures. This could still take some time. STH is still in the process of adjusting the style of the management reporting documents. Although a phone call with JMS has clarified many of the issues, a written example would guarantee that the format is ok. WFAU are working on the access rights to the science archive data. STH reminded us that we need to send in our monthly reports to him before 31st May. <<<< 5. Data archives update UKIRT: this is now back online and fully accessible (hopefully). The data occupies 1.5 Tbytes of space, consists of millions of I-node gobbling tiny files and covers the period up to mid-2004. It has also all been completely re-organised in a simpler and more consistent manner and has been backed up on 5 LTO-II tapes. WFCAM: the data is online and JRL said that the query interface is 75% working. This will look like the ING archive interface and for now have straightforward password protection. At the moment JRL is tidying up the download facility and including the option of retrieving Rice tile compressed or uncompressed data. There was some discussion as to what the period data should remain in the transfer disk buffer. JRL and MJI suggested 48 hours, while STH and RGM suggested 1 week. The problem, as usual, is limited disk capacity in the staging area where the data can be accesses via protected http. It was decided that we start with 1 week and see how things go. It was agreed that there would be a 0.25 TB limit for each request. AAO: most of the data (~1 Tbyte) is now online. JRL will start working on the download side of the access. The user interface already exists. <<<< ING: a start has been made reading the DVDs. However the favoured DVD reader/writer has now expired under the strain and either needs fixing or replacing. ESO transfers: these have not started yet but will as soon as MR returns. 6. Optical processing update - VST surveys news STH reported that the processing of the INT WFC data for Anna Arnadottir had been completed. MJI said that the public surveys for VST have been chosen. The 3 core surveys are ATLAS (Shanks et al), VPHAS (Drew et al) and KIDS (Kuijken et al). CASU are signed up for the processing for the first two projects. The last will presumably be processed by Astrowise. There are also 3 secondary surveys: UVEX (Groot et al.), z<2 census (Oliver et al.) and UKIDSS LAS/GCS (Warren et al.). There will be a meeting at ESO on 22/23 June to discuss these projects. The goal of the meeting is "to coordinate the various teams and maximise the science products of the surveys" and from our point of view hopefully find out more about the data products required by ESO. MJI will attend representing VPHAS since Janet Drew can't make those dates. 7. JAC telecon and WFCAM processing update The last telecon was on 12 May. MJI reported on the issues raised: The project ID for observations standard stars should be UKIRTCAL and CAL for other calibration observations. This has not quite happened yet for standard star observations, but will be done soon. STH said that there should also be a standard flag keyword set to TRUE, however, this also hasn't happened yet. It looks like the 1st month of science data at least will have these problems with headers. MJI pointed out that it is not a CASU responsibility to alter PROJECT ID keywords in the headers so these will remain as received. There are occassionally missing files, up to now for single extensions. Somtimes this occurs when a single DAS crashed and there just wasn't a file. But there have also been a few cases of files mysteriously vanishing before making it to tape. This is currently a low rate (roughly 1 in 10,000) and has been spotted when the MEFs are being created. Various improvements to the checking procedure at JAC (and in Cambridge) are now being implemented to minimise the occurence even further and to ensure no files go entirely missing. The few single extension missing files that do exist have been recovered and included in the on-line data here (thanks to PH at JAC). JRL pointed out that about half a night of data (1/3 end 11th and first 2 hours of 12 April) was taken with 1 detector (#2) not functioning ie. all zeros in the data file. This probably occurred after a reboot of the data system and was not noticed at the time. There are routine procedures for taking twilight flats in all broadband filters and a full set will be generated at roughly weekly intervals. There have been some hardware problems with the summit LTO drives that are causing delays in shipping out tapes to us. So far this has not caused any problems at our end. MJI pointed out that any University of Hawaii and Japanese data will be processed by CASU and they can access this processed data in the usual way from Edinburgh. They will also be able to access the raw data from the CASU raw data WFCAM archive if they really feel the need. The HAWK-I project was mentioned (see the recent ESO Messenger). The NIR camera has a 7.5'x 7.5' field of view using a 2x2 array of 2kx2k Rockwell 3-edge buttable devices for use on the VLT (giving 0.1 arcsec pixel sampling). This is scheduled for introduction at the end of next year. Although the GRASP is still a factor 3 lower than WFCAM if the image seeing is identical, MJI pointed out that the GRASP also depends on seeing**-2. STH said that the processing was progressing well and that the pipeline was becoming more stable. Four nights (8-11 April) of Science Verification data are currently being analysed. It takes about 24 hours to process one night of data on one CPU on one PC. Since we have 8 CPUs available for processing, this timescale is acceptable. The dark frames have been found to be stable over the night. The variation from night to night is about 10 counts in the DC level but the structure is stable at the few counts level. MJI again pointed out that the apparent sensitivity variation in the flats can be as large as a factor of 2. This appears to be a property of the detectors and not the filters. This will make producing a uniform survey very difficult. STH noted that we were still tweaking the reduction strategy, particularly the sky subtraction. 8. Post-FDR VDFS meeting at ESO (PSB's notes from this meeting are available as a separate document). PSB reported that most things got sorted out during the visit to ESO 10/11 May. ESO did not get a chance to review all the documents before the meeting, so some small changes may still be needed. The Board Dispositions are almost all done. One of the changes is that the SDT has now become the SADT (Survey Area Definition Tool or SAD Tool). WJS was asked to acquire the latest version of the SADT document for us. <<<< During a meeting with the DFS group and the DICB chairman (Adam Dobrzycki) it was suggested that the keyword for observations of standard stars be changed from SCIENCE to CALIB. It was originally thought that since science could be carried out with these observations this would be a reasonable descriptor, however due to issues related to ESO release of data CALIB is preferred. This needs to be reviewed by the VISTA project as a whole. A new opportunity for pipeline processing at Garching was revealed, a super-recipe. The processing for this is not triggered at the template level, but by someone "pushing a daily button". This idea might involve CASU having to provide some more recipes but it enormously simplfies the maintenance of calibration files. JRL and PSB were given a CPL tutorial which went well. It was pointed out that we will have to deliver v0.1 of the DRL by the end of the year and the ETC C modules. 9. Plone for documentation EGS said that the transfer of the Plone system has been fully completed from apm14 to apm15, which has two processors. Backups are done nightly and are kept for one week. If apm15 goes down apm14 can take over. DWE said that he had spent some time exploring the system and although there were many useful features, the overall impression was some simple tasks, like document writing, were harder. EGS said that one of the problems encountered by DWE (the bulk transfer of plots) could be overcome by using FTP. STH said that he liked it, but that it did need some reorganizing since he found it difficult finding things. He also didn't like the name "Issue Collector". Of the forthcoming suggestions, the use of Bug and Fault were rejected since the this tool covers more than this. In the end it was decided to rename it the Reporting Database. <<<< PSB asked what the Gaia group were intending to do with Plone. DWE said that Plone had been installed on Vasily Belokurov's Linux machine and that he was in the process of setting something up. 10. AstroGrid liaison update Clarification was provided on the status of cass123. This machine has now been physically moved into the dungeon. GTR/STH had directly contacted Hardip Sanghera regarding the managing of cass123. It was agreed that the system managers would continue to manage and upgrade the Solaris side of this machine and that the AstroGrid software would be managed by STH. The only additional work for the system managers would be to add cass123 onto the backup schedule. PSB said that the stated action had been very woolly and was not clear as to what exactly the system managers were being asked to take on. In order to make sure that everything was clear, PSB and STH will discuss this further. <<<< RGM said that he would distribute a document by Tony Linde on service support for VO. <<<< RGM also asked if we needed any more hardware for AstroGrid purposes. There is a small amount of money available, but it must be spent by July. RGM, PSB and STH will have a meeting to discuss what should be bought with this money. <<<< 11. AOB - GAIA news STH said that AstroGrid finally gave their presentation to the Staff Meeting and that they were generally impressed with it. The questions asked were mainly to do with how would it be supported and will any other system duplicate its functionality. DWE reported on a recent Gaia group meeting in which Gerry Gilmore reported on his presentation to the PPARC science committee. This was apparently well received and the committee seemed to be extremely enthusiastic about Gaia. MJI said that the heating problems in the APM building have still not been sorted out but that he has found a way to temporarily override the problem (too smart central heating and boiler controllers) Continuing Actions ------------------ JRL include access to "observing" log index files in raw WFCAM archive MJI produce a new report on the detector properties as they currently are JRL and illustrate with processing examples EGS MJI organise producing M51 colour picture needed for publicity New Actions ----------- MJI find out who will be attending the UKIRT Board Meeting MJI place the WSA Photometric Recalibration Design Document on our internal web page ALL read and provide comments on above document EGS collate above comments ALL send in monthly reports to STH by 31st May JRL progress download side of AAO archive scripts WJS acquire the latest version of the SADT Definition Document MR rename the Issue Collector in the Plone system to Reporting Database EGS PSB make sure all responsibilities regarding cass123 are understood STH RGM distribute document by Tony Linde on service support for VO RGM have meeting to discuss what equipment needs to be bought from the PSB AstroGrid money STH