23th March 2006 Observatory meeting room 10.00 - 13:15 Present: DWE, MJI, PSB, STH, MR, JRL, RGM, EGS Apologies: NAW, WJS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. VDMT and Q1 reporting and WPs 4. Data archives update 5. Optical processing 6. WFCAM processing 7. VDFS VISTA update 8. ProtoPlone website 9. AstroGrid liaison update 10. AOB Minutes ------- 1. Actions from last meeting JRL has not received any news about the OMP database mirror and its repair. He has emailed Andy Adamson about this. The script which updates the MSB quality flag has been written. JRL will send this to MJI for inclusion in the pipeline. <<<< MJI has made no further progress investigating the merging of classification statistics - ongoing <<<< JRL has not finished the mods to the AAT archive but will by the end of month. It was pointed out that the AAT board will meet 3/4 April. <<<< STH has circulated a set of dates for the next VDMT meeting which is in the process of being organized. STH have reassessed the 06Q1 work packages and progress in advance of the MJI VDMT meeting. MJI said that he would circulate the ASCII version of <<<< this and we should all send back comments to him. <<<< WJS tried to sort out a CASU visit to RAL during AIT3b (see item 7) PSB PSB made a list of desired AIT3b data and sent this to RAL (see item 7) WJS MJI has raised the issue of requirements on CASU during VISTA commissioning and is awaiting some documentation regarding the plans for this MJI were unable to include pointers to tables within the ETC help pages PSB since they are still waiting for digital data. This is subsumed into the ETC workpackage. Meanwhile it was suggested that WJS expedite acquiring the necessary tables. <<<< PSB checked the ETC and it uses the current set of numbers that WJS has provided ALL have read the VISTA survey proposals to check for possible science involvement and unexpected processing requirements. A small meeting was held to discuss this. MJI updated JPE's data management plan template. CASU as a group is normally listed and only where appropriate are named individuals mentioned as Co-Is. MJI has not contacted JPE regarding CASU preferred method of data transfer from ESO - ongoing <<<< RGM is working on quantifying the foul-up causes in LAS outliers list - ongoing <<<< DWE added that 95% of detected persistent images are classified as galaxies and 5% as stars. ALL had read the new version of the EDR paper and sent in any appropriate comments (see item 6) ALL had a look at the new Plone site and sent in comments to DWE and EGS (see item 8) JRL has upgraded the WFCAM pipeline so that it will cope with completely missing detector data MJI is providing input for the VST SMPs STH has not found out what happened at the AstroGrid Implementation meeting. He pointed out that no minutes had been published - ongoing <<<< 2. Comments on WFAU minutes The main topic was transfer speed between CASU and WFAU. In general this is 10 Mbytes/s, but seems to vary between 3-14 Mbytes/s. CASU are still puzzled by the requirements on transfer speed. All of WFCAM 05A and 05B processed data occupies only 16 Tbytes of disk space. A 10 Mbytes/s sustained transfer speed would enable the complete set to be copied in less than 3 weeks. CASU are still concerned where the bottleneck is and PSB was asked to get MRTG set up on apm7 to monitor the traffic. <<<< It would also be useful if WFAU could provide some information about where they feel the bottlneck in network traffic is when they detect low transfer rates eg. some traceroute stats. The costs of UKLight were also discussed. MJI is concerned about the recurrent costs of UKLight for which there appear to be no figures available. It was noted that all of the relevant WFCAM O5A and 05B internet transfers to ESO are complete and that as this is controlled from a different machine it would have had negligible impact on transfers to WFAU. There was also some discussion regarding the blitz nature of the transfers which renders apm7 almost unusable when they are in progress. 3. VDMT and Q1 reporting and WPs MJI said that the forward plans for Q2 have been produced and will be circulated for comments (see above actions). STH requested that we all send him reports covering the whole of Q1 by the end of the month, even though several people have already sent earlier monthly reports. <<<< 4. Data archives update AAT: JRL said progress had been made on the AAT archive interface. He was reminded that it was important to get this finished by the end of the month. WFCAM: up to date UKIRT: up to date. ING: although most of the data (apart from small gaps) is on disc, the current archive setup only knows about which CDs to be read since the archive does not know the location of the online data. This is not completely straightforward due to duplicate entries which need culling. There is also the issue of reordering the earlier data using date subdirectories. JRL pointed out that this will only work back to about 1990 since earlier data often had no date information in the header. EGS has made a start sorting out the duplicate entries and he, JRL and RWA need to get together urgently to sort this out. <<<< 5. Optical processing MJI reported that a draft of the VST public surveys Survey Management Plan for the 2 UK-led VST surveys was being written. The VPHAS+ plan should be finished by the end of the month and will be used as a starting point for the ATLAS plan which is due two weeks later (sic!). 6. WFCAM processing Photometric calibration status: STH said that since the last meeting a preliminary investigation into the Y and Z calibration has been carried out using the Cohen standard. MJI was concerned that in the random check he did for this object it was flagged as saturated in the Y and Z catalogues and therefore BAU. Alternatives are being investigated including spectrophotometric standards from Sandy Leggett, some UFTI data, and a comparison with SDSS z' data. The updated WFCAM calibration will use a restricted 2MASS J-K colour range of 0.0-1.0 to help exclude late-type giants, weird objects and heavily reddened stars. This should produce a more stable calibration for the Y and Z bands (but have little impact on the JHK calibration). MJI said that before starting the recalibration he wanted one final look at the colour equations currently in use to see if any small final tweaks were needed. MJI initiated some discussion on what was needed to describe the error on the photometric zeropoint and the nightly variation in extinction. Was more than a single number needed ? Opinion was divided on this. Persistent investigation: DWE was asked to get the persistence report ready for external release. <<<< He was also asked to add a few examples of persistence to the examples page and include frame number and X,Y position on chip. <<<< UKIDSS EDR Sci and CASU papers: STH said that the EDR paper was already on astroph. The group thought that Simon Dye should be congratulated on doing a good job in producing the paper. MR reported that a good start on the draft (13 pages and rising) of the CASU technical paper had been made. MJI thanked MR and EGS for progressing this and noted that he would work on the draft after the ESO deadlines. Problems eg. UDS: MJI mentioned that he had received an email from Omar about problems with the EDR UDS catalogue data and asked if anyone was aware of other issues with the EDR data. RGM showed some examples of LAS outliers. Some of the problems seemed to be caused by the deblending algorithm. JRL said that he would look into some of these examples. <<<< Reprocessing 05A: The reprocessing of the 05A data will take place soon. JRL said that the main change was in the determination of the sky (and also cross-talk correction would be applied). The selection of which frames are used has changed and the object rejection within each contributing frame has changed. During the discussion it became apparent that avoiding the list of bad frames forwarded by Nigel (those not to be used for sky determination due to Moon problems) had not yet been implemented. MJI said that he would <<<< email this (again) to JRL who would then implement it. <<<< There was also a raw data header problem list of files that had slipped through the fixing net. This also needs dealing with. MJI noted that <<<< he had already fixed the DXS (and a few UDS) QUICK LOOK recipe keyword problems. Extra quality control of the reprocessing was discussed and it was agreed that MR and MJI would look hard at the data from 8/9 April to confirm that the reprocessing was satisfactory. <<<< News on science with WFCAM data: STH said that a paper on alpha Sco was in production. This is a GCS target and the pipeline has worked very well with this region. RGM mentioned that he had been doing work on high redshift QSOs and also very red objects. 7. VDFS VISTA update VIRCAM Status AIT3b: PSB reported that he had forwarded a list of requested observations to be carried out prior to VIRCAM shipping. Gavin Dalton noted that he thought that they could do the AIT3b relevant ones at the end of the week, but then having had a quick look at the TCS simulation setup viz-a-viz the camera setup, felt that this was going to be difficult to achieve by the 24th March. He also requested that CASU not attend AIT3b, and thinks that the best time to actually visit is definitely going to be during all-up testing. The tests that have been requested can be at http://www.ast.cam.ac.uk/vdfs/docs/vista/cambridge_ait_request.txt VDFS docs status: PSB said that there had been no movement on the status of the VDFS documentation since the last meeting. DRL status: JRL has been working on the DRL and supporting documentation. A telecon with Sandra Castro and Pascal Ballester was held on 9 March to discuss a dry run of the pipeline recipes on ESO's new infrastructure (Condor). To progress this a meeting in ESO will be held on 26 April with Sandra and Pascal. ETC status: PSB said that there had been 3776 hits on the ETC and people still seem to be using it. Jakob Vinther has been assigned to write the ESO version of the ETC. PSB has sent him the most recent version (v1.1) of the ETC document together with the associated data tables and C code. Summary of VISTA survey proposals: RGM said that there were 14 proposals (known to us) and that this was about a 2-3 times oversubscription factor. He circulated a summary of the known proposals. 8. ProtoPlone website There was much discussion on this, but apart from minor comments (failure of print button and font size was too small), the main topic was the dumping of documents. STH said that the dumping of documents tended to create a mess. At least with Plone, these would be indexed and it would be possible to find them. PSB said that he still wanted an ordinary file system that could be accessed over the web. RGM and MJI did not see why we couldn't have a hybrid system. STH was very much against this since it would lead to confusion as to the location of documents. JRL said that we should give Plone a go in order to see where the problems were etc......... DWE was concerned about trying to maintain two systems. MJI suggested that a script be written to check what files had been recently added to the file based system and automatically load it onto the Plone system. DWE said that doing it automatically wouldn't work due to different organizational structures, but he would look further into this. <<<< RGM and STH clarified a lot of the discussion by stating that there were two fundamental issues 1) how do we handle our internal documents 2) how do we organize our public interface. The latter was probably the more important. MJI asked DWE to reorganize the internal file system on the same basis as the internal Plone system. <<<< 9. AstroGrid liaison update Nothing to report. 10. AOB NAM: EGS said that he had not been allocated an oral presentation for the "New Techniques & Future Facilities" session at NAM. He will now contact Steve Warren to try for the "Wide-Field Surveys: First results from UKIDSS" <<<< session. It was felt that if we can't get an oral presentation then we would eschew presenting a poster. MJI drew our attention to an image processing workshop in Croatia. The deadline is the end of March. He said that he would distribute the details by email. <<<< IoA Open Day: we will not be opening up the APM building. Continuing Actions ------------------ JRL provide a script to add JAC MSB quality flag to the FITS headers MJI finish investigating merged classification statistics JRL finish mods to AAT archive interface by end of March MJI contact JPE regarding CASU preferred method of data transfer from ESO RGM finish quantifying foul-up causes in LAS outliers list STH find out what happened at the AstroGrid Implementation meeting New Actions ----------- MJI circulate the ASCII version of the 06Q1 work packages ALL feedback comments regarding the 06Q1 work packages WJS contact JPE regarding acquiring the numeric (rather than graphic) versions of the data that is required by the ETC PSB set up MRTG on apm7 to monitor network traffic ALL send reports to STH covering the entire quarter JRL have a meeting with RWA to expedite progress with the ING archive EGS MJI recalibrate the colour terms of 05A STH check if the colour terms need updating DWE get the persistence report ready for release DWE add a few examples of persistence to the examples web page (including frame number and X,Y position on chip) JRL investigate deblending problems identified from LAS outliers list MJI email Nigel's list of bad sky frames to JRL again JRL implement bad frame list in sky determination procedure JRL check and if possible fix any remaining FU'd headers MR check reprocessed data from 8/9 April for quality control MJI DWE write script to check for newly added files to internal file system DWE reorganize the internal file system cf. Plone internal system EGS contact Steve Warren regarding giving a presentation at the Wide-Field Surveys session at NAM MJI distribute the details of Croatia workshop