31st August 2006 Hoyle Committee Room 10.00 - 13:00 Present: DWE, MJI, JRL, PSB, MR, STH, EGS, RGM Apologies: WJS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. News from IAU GA 4. Upcoming VDMT meeting 5. CASU help saga 6. Data archives update 7. Optical processing update 8. WFCAM news 9. VISTA news 10. AOB Minutes ------- 1. Actions from last meeting JRL has not implemented the fix to the problem with the UKIRT archive (missing header information problem) - ongoing <<<< STH have had more than one meeting to plan the update to the Calibration PSB Plan document (VIS-SPE-IOA-20000-0002). MJI DWE have had a number of meetings and have a draft version of the Plone EGS site ready for reviewing by the rest of the group. DWE will send out STH a reminder of the address. Please send comments to DWE and EGS. <<<< MR MJI emailed JAC to see if it would be possible to exclude daytime engineering test data from the data sent to Cambridge. This is being looked into by JAC. PSB has set up the casuhelp email account. This is aliased to the wfcam account and the .procmail diverts all emails to MJI. DWE has placed the casuhelp email address on the web/Plone pages STH has not yet acquired the tiled supernova remnant image. JMI did not have access to the data, so he will now try Albert Zijlstra - ongoing <<<< MJI decided not to draw up a response to the UDS EDR paper - withdrawn MR STH has not investigated the effects of stacking on the 2MASS/WFCAM non-linearity comparison - ongoing <<<< PSB have had many meetings to decide on what documents are necessary for the STH UK review JRL MJI STH attended a meeting regarding the Z filter (with JPE and PCH). There was PSB an agreement to have a filter that looks very much like the WFCAM one. It would be interesting to know if this has been ordered yet. DWE forwarded the Gaia meeting email to the rest of the group. ALL the ADASS 2006 deadline for abstracts has passed. EGS has submitted an abstract. 2. Comments on WFAU minutes PSB described some of the transfer tests that have been carried out and is pleased that they have been done. He added that nothing much is happening on the UKLight front, although the Cambridge end is ready to go. DWE raised the issue of CASU web addresses and the need to inform people of any changes when we swap over to the Plone system. There then followed a discussion regarding the transfer period between the HTML and Plone sites. For the first month or so, it was felt that the 2 systems should be running in parallel. The speed of the Plone server was also discussed and it was decided that a dedicated server should be bought. EGS, PSB and MJI will write a specification for this machine. <<<< It was also decided that this machine would host the ING archive and possibly other web-based services. DWE asked for more details regarding the flagging of persistent objects. MJI said that he, SJW and NCH were considering what needs to be done as part of a general discussion on flagging "bad" objects. MJI mentioned that there had been problems in sending a header updating script to WFAU. The problem was caused by the attached script being embargoed for security reasons and then redirected to a black hole, without informing either recipient or sender that this had been done. The solution was to send the Perl script gzipped. 3. News from IAU GA DWE said that the IAU assembly had been very interesting including some entertaining planet-definition meetings. Another measure of flux for extended images was mentioned by Konrad Kuijken during one of the talks (psf-corrected aperture-matched colours). MJI said he was aware of this measure and had discussed it with Konrad at a VST public survey meeting last year. The Commission 8 meeting was more interesting than usual since it was dominated by new telescopes and space projects rather than results from outdated equipment. An extensive document explaining the most recent IAU resolutions was mentioned (USNO Circular 179 by Kaplan). Pat Wallace gave a summary of this which will hopefully by available from the Commission 8 website (which we will be hosting for the next 6 years due to DWE being the new vice-president). The most relevant items here are dropping the redundant EQUINOX keyword when using ICRS and the abolition of Sidereal Time (it is replaced by the more fundamental Earth Angle Time). 4. Upcoming VDMT meeting MJI said that there was a bit of uncertainty about the next VDMT meeting and that it might be held on the 4th or 5th September (or postponed by at least a week). No agenda has been distributed yet. 5. CASU help saga MJI reported on the latest emails received to give a flavour of the type of questions asked. 12 relevant emails had been received in the period since casuhelp had been set up. On average each took about 1 hour to deal with to fruition (one email reply is still pending). A number of them covered missing data, or data not yet transfered to WFAU. Others were giving feedback and asking questions about the VDFS and the VISTA public surveys There was then a discussion regarding how visible the casuhelp email address was since none of the above emails went directly to casuhelp. DWE was tasked to add the casuhelp email address to the bottom of each Plone page. <<<< STH was asked to contact SJW to place this email address on the UKIDSS website and advertize it. <<<< 6. Data archives update AAT: JRL reported that AAT had swapped over to use PostgreSQL and that the automatic updating of the archive had thus broken. JRL will fix this. <<<< ING: JRL said that EGS had done an excellent job with the ING archive user interface. This covers 1996+ data and should now be the default interface. If earlier data is required, users should be redirected to the old interface. The new interface is almost ready to go live. <<<< There are still a few things that need fixing in the headers of some of the files with the 1996+ data such as some FILTER and OBS keywords. JRL will write a script for RWA to interactively run on these files. WFCAM: MR reported that all tapes up to 31 July had been received (ie. all of 06A). The raw archive was also up-to-date. ESO transfers have about one week of data to go before they have completed 06A. All the data has also been processed apart from one night (27 July), which is still going after 6 days of processing. This is because it is part of of the GPS survey and is a very crowded field. RGM pointed out that the WFCAM summit pipeline sometimes had backlog problems because of these fields. JRL said that this was due to the requirement of processing the whole magnitude range. For the VISTA summit pipeline this will not be a problem since the requirement is for DQC only and the magnitude range to be processed can be limited. UKIRT: JRL said that a couple of fixes had to be carried out after the Debianizing of the host computer. JRL was asked to email AJA that all should now be OK for routine inquiries. <<<< 7. Optical processing update MJI said that the revised Software Management Plans for the VST surveys were sent in by 10 August. He has also been finishing off processing the 1/2 Tbyte of UVEX data for Paul Groot (a proto-type Northern equivalent for part of the VST VPHAS+). EGS will be re-reducing some data due to a RAID system meltdown (see later). 8. WFCAM news Progress with 06A: As mentioned previously, only one night from 06A is left to process. This night, 27 July, would be a good one to stress test any pipeline. MJI said that he was checking each night and then flagging them as ready to copy afterwards. So far, 23 nights have been checked. MR reported that he was reingesting all the 05A, 05B and 06A data into the DQC database. This will take of order a week to complete. Calibration update: STH said that updated (slightly) colour equations have been derived from all of 05A and 05B and that these have been applied to the calculation of the photometric zeropoints of all data processed so far (05A, 05B and 06A). DWE and JRL asked STH/MJI to email these colour equations. <<<< STH reported that SJW is writing a UKIDSS paper and that from changes in the median colours he is finding an RMS colour variation of 2.5%. MJI suggested matching locii from colour-colour diagrams as a more robust method. ZY zeropoint: STH noted that as there were no standards for these passbands, extrapolation from 2MASS JHK values using linear colour equations was being used. However, the relationship between Y and the JHK bands is both non-linear in colour and luminosity class-dependent. As a result, A0 stars (for example) were offset by 0.1 mag in Y. Thus not all star types are on the Vega system for these passbands. PSF fitting: DWE reported that the tests with real data have gone reasonably successfully. More checking is required with the M17 case although Prague and Dubrovnik have got in the way of this. A new modification has been carried out to the standard pipeline astrometry making it as good as PSF fitting across the entire magnitude range for most circumstances. After some discussion it was felt that there was a need to benchmark the PSF software with respect to other PSF fitting software. STH suggested Phil Lucas as a possible person to contact on this point. It was noted that both Paul Hirst and Sandy Leggett have moved from UKIRT to Gemini. 9. VISTA news VIRCAM status: PSB noted that a VIRCAM shipping readiness meeting was being held at RAL. The dubious chip has now been replaced and there was a possibility that we would get a digital QE response curve for the replacement chip. He again stressed that if there was another opportunity to acquire some test data, it would be very helpful. The possibility of doing this later on Paranal was discussed. Recipe keywords: the VISTA VIRCAM data-acquistion software provides a RECIPE keyword in the FITS header to drive the UK VISTA Science pipeline. This keyword is explicitly ignored by the ESO Quality-Control pipelines but is required for options within the UK pipeline processing. PSB, JRL and MJI need to come up with a proposal for these and discuss with SMB. <<<< VDFS docs status: The ETC has now been updated to have the new overhead algorithms. This results in a ~10% improvement in observing efficiency. JRL said that he wanted to update the data flow figures in the DRLD. STH will provide these. <<<< The VISTA data transfer from ESO is currenty planned to be via disks. These are currently standardized at 250 Gbyte Linux file systems (ext3). The benefits (factor of 10 less disk traffic) of using the latest 750 Gbyte disks with Rice compression were discussed. PSB thought that an ICD on data transfer from Garching to Cambridge was needed. He will raise this issue with JPE. <<<< Upcoming UK review: MJI outlined the documents and web access needed Overview of pipeline processing philosophy MJI Document summarising the documents and discussing the UK URD requirements with comments and pointers to how/where these will be MJI met/discussed or otherwise (may include addendum of VISTA public STH survey requirements) Detail of the pipeline processing (aka latest Calibration plan and DRLD) JRL which covers calibration and pawprint processing PSB Extra document in same vein describing additional processing operations for UK processing eg. mainly tiling/mosaicing and interpolation options JRL Hardware design document including proposed data transfer mechanisms, hardware for processing and storage system PSB DQC and pipeline progress database and web pages MR EGS More detailed discussion document for photometric calibration based on STH WFCAM strategy. Draft of WFCAM processing technical paper (as example) MJI Plone website including technical description of WFCAM processing etc.. DWE examples of progress web pages etc... as examples EGS MR 10. AOB ADASS 2007: JRL said that he wanted a meeting next Monday to discuss organisational details for ADASS 2007. Hardware update: MJI reported that the equipment grant (April 2005) has finally made it, so we can now sort out terminating the old grant and assessing how we stand with equipment procurement. The solaris UFS file system /data/cass55c was totally fsck'd and all data lost. This RAID array has now been moved to the linux machines and rebadged as the XFS filesystem /data/apm23_c (7 Tbyte RAID5). It is unclear what caused the complete meltdown but there appears to be nothing wrong with the disks. Another 10 Tbyte RAID6 array has been acquired and mounted as /data/apm23_d. Power Cut: To prevent any problems from the mega power outage scheduled for the weekend of 16/17 September, all CASU machines will be switched off late on Friday 15th and isolated from the mains for the duration (a generator will be supplying temporary power over some of the w/e and we don't want that do we ?). They will be re-connected on Monday 18th, or thereabouts, depending on whether or not we have a stable power supply again by then. However, this will not be the end of the saga and sometime after this, but in the very near future, the power will again effectively go off for a whole w/e. Continuing Actions ------------------ JRL finish the fix to problem with UKIRT archive (missing header information) STH acquire tiled supernova remnant image STH investigate effects of stacking on 2MASS/WFCAM non-linearity comparison New Actions ----------- DWE remind ALL about location of Plone site etc. ALL review Plone site and send comments to DWE and EGS EGS decide on a specification for the CASU web server PSB MJI DWE add the casuhelp email address to the bottom of each Plone page STH contact SJW to place this email address on the UKIDSS website and advertize it JRL repair AAT archive (PostgreSQL problems) EGS link new ING archive interface to current ING archive page and JRL make it the default with option to use previous interface for pre-1996 data JRL email AJA about fixes to UKIRT archive MJI email new WFCAM colour equations to DWE and JRL STH PSB agree on a proposal for RECIPE keywords and discuss with SMB JRL MJI STH provide JRL with new figures for DRLD PSB email JPE about ICD for Garching-Cambridge interface