26th April 2007 Hoyle Committee Room 11:00 - 13:25 Present: DWE, STH, JRL, PSB, MR, MJI, EGS, NAW Apologies: RGM Guest appearance: Robert Greimel (RS) Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes - UKLight saga update 3. NAM news 4. Report from Second Generation VLT Instruments meeting 5. Data archives update - AAT, ING, WFCAM, UKIRT 6. Optical processing update - IPHAS release 7. WFCAM news - raw data status - processing status - photometric calibration update 8. VISTA news - pipeline progress 9. ADASS 2007 - latest 10. AOB - upcoming VDUC meeting - cass38 and cass36 - rolling grant application Minutes ------- 1. Actions from last meeting JRL has not repaired the AAT archive mirroring (PostgreSQL problems), however some progress has been made since Stuart Ryder has been in touch. Since this issue seems very much out of our hands, it has been decided to drop this as an action and pursue it as part of the archive redevelopment. MJI did not email Matthew Colless regarding the above problem. This action has now been superceded by the above events RGM has not reported back on the classification aspect of cross-talk weeding. EGS and MJI are looking at classifcation generally and will contribute to this action - ongoing <<<< EGS prepared the ADASS 2007 preliminary announcement web page - MR http://www.adass.org:8080/Conferences/2007/Venue/ MR continues to investigate credit card payments for ADASS 2007 - ongoing <<<< PSB is waiting for a quote for wireless provision for ADASS 2007 from a local (Cambridge area) contact. WJS the status of field rotation flat field tests and VISTA integration is unknown. MJI will raise this issue at the VDMT meeting tomorrow. <<<< MJI noted that WJS has now officially left the IoA as of 31st March and all wished him well in his new job. JRL has not found out more about the previous providers of wireless services at Kensington Town Hall. This action has been superceded by a previous action. RGM has emailed Tim Naylor, JPE and WJS about discussing the updated VDFS UK User Requirements Document as part of the VDUC agenda. MJI said that he would also raise this directly at the VDMT meeting on Friday. <<<< The VDUC meeting will be held in Edinburgh on 10th May. STH said that there will be a UKIDSS Survey Heads meeting in Oxford on 9th May. There will also be a calibration meeting tagged onto this (before or after). NAW said that he would email Tim Naylor to obtain the agenda for the VDUC meeting and the minutes from the previous meeting. <<<< MR has not contacted the Durham group about raw WFCAM data processing progress. It was intended that this not be done until the end of April to give them a clear month to work on it - ongoing <<<< MJI distributed drafts of conference proceedings of ESO calibration JRL workshop MR ALL commented on above drafts MJI will ask about the possibility of obtaining more VISTA test data at next VDMT meeting - ongoing <<<< 2. Comments on WFAU minutes PSB reported that the UKLIGHT connection is almost "up and running" between Edinburgh and Cambridge. Some problems still had to be sorted out after the original announcement and there is still no SSH connectivity. Ping works ok, but physical tests have shown that the signal is at the low end of the spec. There then followed a discussion on fast links to ESO and ESAC (for GAIA). It was felt that making progress on a fast link to ESO is currently akin to flogging a dead horse. However, NAW and PSB agreed to discuss the ESAC link possibilities offline to see how they might progress this angle. <<<< STH noted that WSA had installed a new toolkit and asked if this included list driven photometry. MJI said that it did not since that has not yet been supplied. It was also noted that DR2 for GPS had now been released. 3. NAM news EGS, RS and NAW attended NAM last week. EGS reported that Michael Rowan-Robinson had said that photometric redshifts "don't work for the WFCAM filters in the DXS, but that it did for 2MASS data". This was regarded as odd since WFCAM photometry is calibrated from 2MASS and the colour terms are small. EGS said that he will look into this. <<<< Also mentioned at the meeting was that the funding priorities for STFC (nee PPARC) will be for science-based projects that are UK-led. 4. Report from Second Generation VLT Instruments meeting JRL and PSB reported on the 2nd Generation Instrumentation Pipelines meeting held at the Max-Planck-Institut fur extraterrestrische Physikon 18 April (http://www.mpe.mpg.de/ir/meetings/vlt2gi.php). Many instrument groups were present and they almost outnumbered the ESO personnel present. Many CPL issues were discussed. One of the main criticisms of CPL is that you can't update a file - you have to completely rewrite it. QFITS will be replaced by CFITSIO in version 4.0 of CPL and access to WCSLIB will be included. This FITS i/o change will not make much difference to our software deliverables since the QFITS layer is hidden underneath the CPL interface. This change may open some interesting possibilities regarding use of Rice tile compression. NAW asked about Reflex (lead by Richard Hook). This is a modular system unlike the monolithic recipe-based CPL approach. It was noted that this is only funded until November and will then be reassessed. 5. Data archives update AAT: JRL reported that we have recently received some more data (on DVDs) which has now been read onto disk. However, it was noted that this archive is currently offline due to problems with mirroring the AAT end of the data base (which mainly contains an ingest of the header information). UKIRT: JRL said that more data has just been received and that it is in the process of being ingested. ING: MJI reported that internet transfers from La Palma are proceeding routinely. RWA has gone to La Palma to try reading in the remaining 300 or so early vintage (VMS backup) tapes that are causing problems with our tape readers. WFCAM: MR said that all the raw data (all chips) has been read in up to the end of 5th April except for 6 nights in January and a few other possible problem nights with apparently missing files in November. These are still being chased up. JRL mentioned that there is a group keyword problem with some of this early 06B data. PSB is carrying out FTP tests with Brad at JAC and obtaining 64kBytes/s with a single-threaded transfer. This seems to be a software limit, since multiple FTP sessions can be set up, each getting the same data rate. It is likely that a solution can be found to increase this. Even with an increase to the "wire limit", this may be much slower than say internal UK transfer and may only be useful for occasional transfers eg. missing files. Investigations are still ongoing. Meanwhile PSB was asked to email a copy of his findings to Brad (CC'd to Andy Adamson). <<<< ESO transfers have been carried out up to 24th March with the ocassional hiatus with what appears to be disk changeovers occuring. MR will ask for more details since its a tad inconvenient having to keep stopping and restarting the transfers. <<<< JRL said that there was a press release about the Kognitio data base system that he has tested with 2MASS data. This system uses multiple processors and highly parallelised queries to avoid indexing tables. It is impressively fast at dealing with queries. Kognitio are keen to carry out further tests with the IPHAS dataset. 6. Optical processing update - IPHAS release EGS reported that the data from 2003 to 2005 has been ingested and that the aim is to have a public release on 1 June. The documentation for this is being written by EGS, NAW and RS and a paper will be placed on astroph soon. NAW said that there should also be a press release. There was then some discussion about how the Kognitio system would cope with a merged data base of IPHAS, 2MASS. A complete cross-federation with 2MASS was suggested as an interesting, and useful, challenge. This is definitely worth pursuing given the results so far. JRL said that he would get in touch with Kognitio to see if they would be interested in doing some further tests. <<<< 7. WFCAM news MJI noted that it is time to think about getting some more disk storage space, probably more of the same fibre channel RAID6 systems. There was also some discussion about what to do with the two extra spare 750 Gb disks. EGS said that he could use these as backups for the IPHAS data base. MJI said that he would double check that there were two extra spares first. <<<< MR said that, ignoring data before 20 December and the incomplete 6 nights in January, only a few more nights of 06B (up to 31st March) data remain to be processed. MJI reported that our request to JAC for doubling the dither offsets for twilight flat sequences is now happening. This helps get around the occassional mega bright star screwing up the flatfield frame stacking. MR noted that there are still a lot of incomplete groups and that typically this occurs because the 1st exposure in a group sequence is badly trailed causing the sequence to be interrupted and restarted. MJI noted that we now should be putting the night log comments into the processed directories for completeness. <<<< He also noted that he had received an email from AJA about long exposure (generally NDR) light leaks in dark frames. He and MR will keep a weather eye out for these problems. <<<< STH gave us an update on the photometric calibration. The Y band has been done to +/-1%, but the Z band has not been finalised yet. MJI asked about the consequences of not using SDSS preselection and STH noted that the same numbers were obtained. A meeting of the photometry working group in early May is planned. MJI said that there will be a UKIRT board meeting in late May and that it would be good to keep the web pages containing the relevant photometry calibration information updated since these are referred to in his draft report to the Board. <<<< STH said that investigations have been carried out into Eddington bias. This is apparent when you compare deeply stacked data with shallow survey data due to rms fluctuations biasing fainter than average objects above the detection threshold. However, since we cunningly use a 10-sigma detection threshold for the photometric calibration from 2MASS, the bias at the faint end is only 1% and the overall bias in the derived zero-point is much less than this. This work is still ongoing but the effect, as expected, seems to be negligible. MR was congratulated for his article on the properties of NIR sky brightness in the UKIRT newsletter. During some tests on LAS data in DR2 RGM and EGS noted a small effect (an offset averaging ~1%) when comparing DR2 band-merged products with CASU generated band-merged products. EGS asked if the WFCAM photometry has been corrected to allow for the astrometric distortion. MJI noted that this is allowed for directly when computing the photometric zero-points but is not applied directly to the catalogue fluxes since the principle has always been that the fluxes should just be the sum of the relevant pixel contents. As with the aperture corrections the intent has always been to allow for this on-the-fly when forming magnitudes. The WCS information in the catalogue headers provides the necessary information. MJI noted that the CASU band-merging software does compute these and apply them during the merging process and that this explains the small offset (~1%) discovered between the local band-merged products and those available in DR2. MJI said that he would email NCH about this. <<<< 8. VISTA news JRL said that he had received comments back from Sandra Castro regarding the 0.5 release. The ongoing issue of continuing changes to the content of PAF files was discussed. Various email exchanges have happened to try and put a cap on this by getting ESO to define exactly what they want in the PAF files. Convergence may be an illusion. PSB raised the issue of what the exposure time value -v- NDITS contained within the VIRCAM FITS headers means. MJI said that he would also raise this item at the VDMT. <<<< PSB was congratulated on writing a web page explaining how to put WCS information within FITS table headers, as opposed to FITS image headers, http://casu.ast.cam.ac.uk/documents/vista/astrometry/tables-wcs/view is the path to enlightenment. It was noted that from 07B, WFCAM catalogues will adhere to this format within their headers. Although two parts of llareggub software can interpret this, it is now the agreed FITS standard and does at least pass fverify. We expect WCSLIB to soon adhere to this form and when it does we will adopt it for WCS info i/o. MJI said that an FP7 proposal is being lead by ESO to improve the data link between Paranal and Antofagasta in Chile. JPE is currently at ESO helping to draft this. There was no further news regarding hardware requirement/procurement for Paranal. 9. ADASS 2007 JRL said that 3 proposals for tutorials have been submitted (Data mining, 3d visualization of data, Relational data bases). Thus, there will be a tutorial. Should there be a charge for this? Yes otherwise = chaos. This is the usual precedent and will encourage people who have signed up to turn up. The next milestone is 1st May, when the registration page needs to be ready. 10. AOB MJI said that cass38 has now been retired properly and that cass36 was next on the list. There was some discussion as to what functionality from cass36 needs to be preserved. MJI said that the rolling grant application will be submitted late May and that input was required for various parts of the grant bid. <<<< Use of office space was discussed. Due to the upcoming building work on site pressure on space will be more acute. PSB will be sharing with JRL and this has to happen before September. Earlier than this is desirable since the process of clearing out the SPO will be very disruptive. Further discussions took place concerning use of the "storage" room and main central area, in particular making the central part more functional for visitor use. Continuing Actions ------------------ RGM report back on classification aspect of cross-talk weeding EGS MJI MR sort out how ADASS 2007 credit card payments are to be made MR contact Durham group about raw WFCAM data progress (~end of April) MJI ask about obtaining more VISTA test data at next VDMT meeting New Actions ----------- MJI ask about status of flatfield rotation tests aka VISTA integration MJI raise issue of updating the VDFS UK User Requirements Document at VDMT meeting NAW email Tim Naylor to obtain agenda for VDUC meeting and minutes from previous meeting NAW investigate possibility of fast links to ESAC (for GAIA) PSB EGS investigate photometric redshifts in DXS PSB send FTP test results to Brad (CC'd to Andy Adamson) MR inquire about causes of intermittent nature of WFCAM transfers JRL contact Kognitio regarding further tests involving merged data base containing IPHAS and 2MASS EGS sort out use of spare 750 Gb disks as backups for the IPHAS data base MJI MJI OMP log comments in processed directories, make it so MJI asses impact of light leaks in dark frames on pipeline processing MR STH update web page on the recent photometric calibration developments in lieu of the upcoming UKIRT board meeting (17/18 May) MJI email NCH to remind him about the photometric corrections needed to account for the astrometric distortion MJI raise issue at VDMT meeting of what is meant by exposure time value contained within VISTA FITS headers ALL provide input on various activities for rolling grant application