2nd August 2006 Hoyle committee room 11.30 - 13:45 Present: DWE, MJI, JRL, PSB, MR, STH, EGS, NAW Apologies: WJS, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report from VDMT 4. Report from UKIDSS meeting 5. Data archives update 6. Optical processing update 7. WFCAM news 8. VISTA news 9. AOB Minutes ------- 1. Actions from last meeting JRL has not implemented the fix to the problem with the UKIRT archive (missing header information problem) - ongoing <<<< [JRL noted that the Debianizing of apm12 required a workaround for downloading data from the UKIRT archive.] EGS updated the INT WFC web pages with the latest flat-field and fringe frames STH has not updated the Calibration Plan document. He said that he needed to discuss this with PSB and MJI. His main concern was with the Z and Y calibration. MJI felt that there was no practical alternative to the currently proposed pipeline solutions. The draft of the updated document would be needed by the first week of September. STH, PSB and MJI will have a meeting to plan what is needed and then update the document <<<< DWE have not finalized the content of the Plone site. DWE said that not EGS much progress has happened since the last meeting due to concentrating STH on the PSF work. Small amounts of content have been added - even a page MR from PSB on networking issues. The technical pages are about half completed. MJI noted that the Plone site needs to be up and running by the time of submitting the UK review documents - ongoing <<<< DWE have set up a local ADASS 2007 web site using Plone EGS JRL contacted WFAU to clarify the problems with the catalogue generating software (see later). MJI compared old and new versions of the catalogue software and MR also compared toolkit development version with pipeline version. They found a few gotchas in the pipeline version and suggested fixes (see below) MJI has not emailed JAC to see if it would be possible to exclude engineering data from the data sent to Cambridge - ongoing <<<< JRL investigated the MSB ID checksums with a view for tagging bad MSBs. MJI This complicated system was found to be not suitable. The ball is now back in JAC's court. JRL has fixed the pipeline problems (see above) MJI raised the issue of VDFS document status at the VDMT meeting. JPE said that he was waiting for the final installment of the SADT document to be completed and then all FDR documents could be released at the same time. JRL arranged Debianizing apm12 with JMI who did ALL sent in quarterly reports to STH PSB sent in AIT and DRL reports to STH JRL PSB ordered the local link from the Computer Service for UKLight connection STH circulated the SJW email regarding "deprecations" 2. Comments on WFAU minutes PSB noted that AstroGrid had provided money for WFAU for hosting SDSS DR3. PSB also noted that UKLight work was progressing. MJI said that he had asked for the transfer of 06A to go on hold until the UKLight connection was up and running, providing it does not take too long. 05A version 3 was currently being transferred, and used for various timing tests. STH said it was to be welcomed that there would be access to the flat files from WSA. Access to these will be available in about a week to the whole ESO community. He also noted the agreement to release the observations of standard stars through the database. STH noted that WFAU are working on a scheme to allow efficient photometric recalibration updates from CASU-held catalogues. MJI pointed out that when the dust has settled on the Y,Z calibration we will re-calibrate all of 05A, 05B and 06A. His hope is that WFAU will then just transfer the updated header information and apply it to their own catalogue and image versions. 3. Report from VDMT MJI and STH attended the VDMT meeting in Edinburgh on 24 July. MJI reported on the main points: The UK review of VDFS will be held in Cambridge on 24/25 October. The documentation needed for this will have to be ready a month before that and this implies that drafts need to be ready a month before this ie. start of September. PSB reminded us that the UK URD had still not been officially released. MJI added that now we know what the main public surveys on VISTA are likely to be we need feedback from the survey PIs on the requirements embedded in the UK URD. An email from SJW to JPE had raised issues concerning sky subtraction problems. Some of these were caused by observing too close to the moon (sic!) but there were still a subset of science frames where the sky subtraction was not optimal. The latest, and hopefully final, version of the sky estimation alogrithm appears to have addressed the main issues and was run on 05A version 3 and is being used in the current 06A processing. While sympathising with the problems caused by poor sky subtraction, MJI pointed out there is a practical limit to how well an automatic pipeline can be expected to do and that it will never be perfect. A further issue raised was the response of CASU to external email queries. This was discussed at a previous meeting (Item 7 060612). Two possibilities were discussed: email MJI and he either deals with them directly, or allocates someone to deal with the task; or we set up a dedicated bug tracking system. NAW suggested OMII-UK and DWE suggested the Mantis Bug Tracker (as used by Gaia). There was also the IoA's helpdesk system written by JRL. STH said that the acutual software was not important, but rather the lines of responsibility and keeping track of the progress. In the end it was decided that since the numbers of these emails is quite low we would start with a much simpler system and revisit this issue in a few months. PSB will set up an email account (casuhelp) which will <<<< start life as an alias for MJI's account. MJI will then allocate someone to deal with the task. MJI asked that any responses to these should be CC'd to casuhelp where they will be saved for future reference. DWE will make sure that this system is clearly visible on the web (and Plone) pages.<<<< The concerns over staffing VST processing were discussed and Tom Shanks joined in via telcon to discuss the shortly to be resubmitted Survey Management Plan for ATLAS. It was noted with some concern that the money for VDFS development runs out in September 2007, well before the pipeline and archive will have had a good shakedown with VISTA survey data. Behind the scenes lobbying is in place to find a solution to this and to ensure a smooth transition to VISTA operations. 4. Report from UKIDSS meeting MJI and STH attended the UKIDSS meeting in Edinburgh on 25 July and gave presentations. These are available from our web site. MJI said that MR had updated the processing status pages to reflect the current state of the pipeline. The main moans came from the UDS folk, with some justification. The recent improvements to the catalogue generating software should go some way to alleviate this problem but this still leaves open the question of how to best form the deep stacks. Sebastien Foucaud and Omar Almaini said they had spent the best part of 2 months fiddling with sky background and excising dubious chunks of data before producing the final deep stack. STH reported that in SJW's summary of survey progress (see presentation on http://wiki.astrogrid.org/bin/view/UKIDSS/Jul06Meet) and noted that DXS was rejecting 38% of the data for deep stacks due to having rather generous sky and seeing constraints. There has been a call for proposals for UKIRT beyond the end of the current UKIDSS 2 year plan in response to the Ellis review. This includes new surveys and even new instruments (sic!). The one new survey all present agreed on was the idea of a Northern hemisphere survey. 5. Data archives update AAT: No news is good news ING: JRL and EGS reported that the last 10 years of data (1996-2006) had now been ingested into Sybase. The frontend of the new archive interface was now being written and should be at the beta-test stage by the end of the month. EGS said that duplicate data was now sorted and could be removed (which MJI duly did having first backed it up). WFCAM: MR reported that tapes up to 13 July have been received, MEF'd and ingested into the raw archive. ESO transfers have been completed up to the end of June. It was noted that WFCAM had been taken off the telescope on 31 July. The frequency of occurrence of nulled detectors remains at the same low level and appears to be mainly at the start of nights. UKIRT: No further news 6. Optical processing update MJI said that he had been processing 12 nights of data from the INT for the UVEX project which is a precursor to VPHAS (VST). This amounts to about 1/2 TByte of u, g, r, i, He and Halpha data (ie. the equivalent of a few pro-rata months of VPHAS+ data. According to ESO's survey-to-FTE conversion he can now take the rest of the year off. STH was very excited about an (astronomical) image he had seen recently displayed as a poster. This was a tiled image of a supernova remnant taken within the IPHAS project on the INT. He thought that we should have a copy of this poster, since it was very striking. He will look into this. <<<< 7. WFCAM news Catalogue software: MR and EGS have been investigating the problems reported by UDS with the cataloguing software in a recent paper. The UDS EDR stack was acquired and the latest version of the CASU software was compared with the results from Sextractor. The end result is that the two catalogues now agree on detection levels and star/galaxy separation and the resulting colour-magnitude diagrams are similar. MJI said that the three main improvements to the catalogue software were: - being able to cope correctly with zero or negative sky - removing the quantization in the noise estimation - improvements to the matched filter for image detection. A consequence of the latter is that another input parameter is needed. Both the pipeline and toolkit versions of the programme now produce results that agree with each other. STH asked if we needed to make a response to the UDS EDR paper (currently on astro-ph). EGS said that there was also another paper making similar comments. It was decided that MJI and MR will deal with this issue making sure that any email is CC'd to SJW and AL. <<<< Non-linearity: STH wondered if stacking could be linked with the magnitude scale difference seen in the 2MASS comparison. STH said that he would follow this up. <<<< Progress with 06A: Processing of the 06A data is complete to the end of June. PSF fitting: DWE reported that the bug in the split PSF fitting routine had been found and that the tests with simulated data were successful. Tests with real data were in progress. Offsets: MJI raised the issue of inconsistent offsets within dither stacks and asked if a sort of "majority vote" could be applied to produce consistent offsets for all detectors. 8. VISTA news VIRCAM Status: No news DRL status: PSB said that the main work that had been carried out over the last few weeks was sorting out the interleaving part of the main science recipe. He had also been in contact with Sandra Castro regarding benchmarking tests to enable ESO to specify the hardware requirements for the summit pipeline. The cataloguing module has been updated in line with the changes to the WFCAM pipeline. The next task is to deal with dead chips. VDFS docs status: PSB said that we are still waiting for the overhead algorithm for the new ETC. His new document describing the data that has been used for the ETC has been sent to the VISTA Project Office to see if they have any objections. There was some discussion regarding the file size of a single tile (0.75 GBytes) and the fact that an interleaved tile would thus be bigger than 2 GBytes and thus cause problems for some file systems. Upcoming UK review: For the UK Review (Cambridge 24/25 October) various documents and access to our plone web site will be needed. There was some discussion as to what documentation was needed. PSB, STH, JRL and MJI will have an offline meeting to plan this. <<<< It was noted that JPE will be visiting Cambridge next week to discuss the specification of the VISTA Z filter with Paul Hewett. It was suggested that STH be present. <<<< 9. AOB Hardware update: apm12 has been Debianized and its existing RAID5 system converted to RAID0 since this is effectively used as processing scratch space. The advantage is that the I/O performance is improved by a factor of 2. If this makes a significant difference to overall processing speed, then the other RAID arrays used as temporary data buffers by the processing PCs will also be swapped over to RAID0. MJI noted that another 10 TBytes of fibre channel RAID6 longer-term storage had been ordered. UKLight saga: PSB reported on the Edinburgh end plans for the UKlight link. Further transfer tests have been carried out, the most notable of which were single-threaded transfers which averaged 16 MBytes/s. Further tests are being carried out and a summary will be collated. Gaia: MJI said that there will be a Gaia meeting of CU5 (Photometric Processing) in Cambridge on 25th and 26th of September. More details can be found on the relevant GaiaWiki page: http://www.rssd.esa.int/SA-general/Projects/GAIA/wiki/index.php? \ title=CU5:_Photometric_Processing:_Meetings:_CU5M02 Note that you will need a Gaia username and password to access this. DWE, MJI, PSB and NAW already have these - any problems get in touch with DWE. In the meantime, DWE will forward the email announcement to those who were not on the original distribution. <<<< ADASS 2006: EGS pointed out that that the deadline for submission of abstracts to ADASS 2006 was August 11 - so get writing <<<< Continuing Actions ------------------ JRL finish the fix to problem with UKIRT archive (missing header information) STH update the Calibration Plan document (VIS-SPE-IOA-20000-0002). Have PSB meeting soon to plan the document. Draft by start September. MJI DWE finalize content of Plone site by end of September EGS STH MR MJI email JAC to see if it would be possible to exclude engineering data from the data sent to Cambridge New Actions ----------- PSB set up casuhelp email account DWE ensure visibility of above email address on web/Plone pages STH acquire tiled supernova remnant image MJI draw up a response to UDS EDR paper MR STH investigate effects of stacking on 2MASS/WFCAM non-linearity comparison PSB organize meeting soon to decide on what documents are necessary for UK STH review JRL MJI STH attend meeting regarding Z filter (with JPE and PCH) DWE forward Gaia meeting email ALL make sure ADASS 2006 abstracts are submitted