6th November 2007 Hoyle Committee Room 10:00-12:50 Present: STH, EGS, PSB, MR, MJI, DWE, RWA Apologies: JRL, NAW, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent meetings/visits/telecons 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical processing update 6. WFCAM news - raw data transfer status - WFCAM network transfers status - processing status - reprocessing status - photometric calibration status 7. VISTA news - pipeline progress - camera test data - letter to PIs - science verification 8. ADASS 2007 9. AOB - windows machine++ Minutes ------- 1. Actions from last meeting RGM have not completed the work on the classification aspect of cross-talk EGS weeding and classification generally. It was decided to subsume this MJI action into general checking since WFAU now flag potential cross-talk affected objects with an error bit code. JRL are still working out a scheme for doing header updates for ING archive. EGS It was decided to remove this from the actions list as it is really a work package. MJI is now forwarding the weekly email JAC reports from Chris Davis to a wider audience. It was also pointed out that these summaries are available on a wiki page. MJI will forward access details. <<<< MJI has checked the status of the reprocessed UDS data (see later). PSB has contacted Brad Cavanagh at JAC for an update regarding the WFCAM network transfer (see later). MR figured out the best to way to advertize the illumination correction information and will now implement this scheme <<<< JRL has finished the self-flatting tests on VIRCAM data and emailed a summary of points to note from the VIRCAM tests to the participants of the telecon. JRL the compilation of a wishlist for the late October VIRCAM cold camera PSB tests was dealt with during a telecon, ie. run dark current template, linearity template, and acquire one full r/o with TCS running to test headers etc.. However this was later subsumed by events. PSB reacquired the helpdesk laptop and PDF'ified the ADASS presentations DWE is now updating the apm1 Windows machine regularly ALL considered attending the upcoming UKIDSS meeting in December. Currently, EGS, MJI, STH and JRL have registered. PSB was still thinking about it subject to availability of dunkelbier. MJI has bought a handheld vacuum for the APM building. This should only be used for computers. PSB investigated getting wireless for the APM building and will go get one This would also be an opportunity to acquire a CASU order book from Paul Aslin. <<<< 2. Comments on WFAU minutes MJI noted the recent problems WFAU were having with reading old LTO tapes and commented that although we haven't had any problems with LTO tapes it was worth doing a specific test to check. MR will to dig out 4 old LTO WFCAM raw tapes and verify, or otherwise, that they can be read. <<<< 3. Recent meetings/visits/telecons Luca Rizzi visited on 22nd and 23rd October to discuss various WFCAM issues and to get to know more about how the pipeline works. A set of notes from the meeting are available and are now on the internal web pages. MJI volunteered to make sure that the various reports to the UKIRT board he has copies of are also available on the internal web pages. <<<< There was a telecon on 9th October mainly to discuss VIRCAM test data reports that JRL had produced assessing the properties of the detectors from the July lab test data taken on Paranal. Much of the discussion centred on the differences between the two readout modes. The CASU wish list for the upcoming cold camera run was also briefly discussed. STH reminded everyone of the upcoming VDMT meeting and the need to get reports in on time. 4. Data archives update ING archive: The old DAT tapes will be ditched. At least one of the cabinets will be kept in the IoA/Greenwich House interface for potential future storage use for LTO backup tapes. RWA will skip the old data tapes when an empty skip becomes available. <<<< AAT archive: The AAT CDs have now been incorporated into the collection in the cabinets in the storeroom. It remains to insert the JKT CDs into the drawers. Another month of AAT data has been copied and ingested via the Internet. Plates: Elizabeth Griffin will visit IoA in the first week of December. Craig Mackay's SPO office has been emptied and will be used to sort the plates from the adjacent plate store. The high resolution Coude plates will probably be sent to Canada and the archive Cambridge plates will go to Laundry Farm in the first instance. Bob will ask whether the PDS machine can be of use in one of the new plate repositories. At Laundry Farm, there are about 1000 direct plates, mostly from US observatories and 4 metre or Schmidt telescopes. About 400 of these plates are objective prism exposures and will go to the new repository in the eastern US. The original Palomar 14 x 14 plates are wanted at Palomar and the fate of the other plates has not yet been decided. Attempts are being made to find a new owner for the POSS 1st epoch plate copies and the custom-built shelving that goes with them. There was some discussion about offering the glass copies as souvenirs at Open Days etc. if all else fails and nobody wants them as a set. WFCAM: MR reported that we have received complete nights up to the 3rd of October. JRL is currently working on a new interface to the WFCAM raw data archive that will allow unregistered users to search and request data that under the current policy is public (ie. older than 18 months or whatever this policy morphs into). PIs will be able to obtain access to data from their project in exactly the same way as before. Tapes are still coming in batches containing different nights for each chip. The mismatch of detector nights and tapes was discussed during the recent visit of Luca Rizzi. Plans to perform direct FTP data transfer tests from JACH to CASU are also progressing and it is hoped to test this later in November. There is a G-bit link from the summit to JAC and it is intended to use an FTP server at the summit to transfer some data directly to CASU after some further tests and discussion on transfer protocols are completed. Transfer of raw data to ESO is still intermittent due to problems at the ESO end. This requires too much monitoring at our end which we would like to minimise so MR will email ESO regarding the raw data transfers to see what the status of their planned improvements is. <<<< STH noted that one of his campaign proposal science frames was officially the millionth observation taken with WFCAM, so presumably drinks are either on him or JAC. MJI noted that each executed MSB now has a unique transaction ID in the FITS header to enable better tracking. UKIRT: no change 5. Optical processing update MJI reported that he had received some positive news via Janet Drew about the status of the VST including a realistic date for when it will begin taking science data. He also noted that he had found and fixed a bug in the INT WFC WCS initialisation software that explains the recent spate of problems with converging on the final WCS solution for detector #4. 6. WFCAM news Processing: We are currently processing September data (note the first 12 nights have no data due to WFCAM engineering work). About half of the remaining nights have already been reduced and the remaining ones are currently being processed. Processing of September data was delayed whilst waiting for another batch of tapes to arrive because we had only three complete nights and an incomplete set of data to form decent flats. However once we received the data covering the whole month of September (on October 26th) it turned out that the flats could not be used to process the data because of an interesting new instrumental feature (see below) that was particularly strong on the nights when most of the flats were taken. The effect manifests itself as a variable channel level "offset" on detectors #2 and #3 and has both a repeatable and a random component, in addition to being strongly, but not completely, anti-correlated between the two detectors. An example of the repeatible part of the effect is available at http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/ 2007b-known-features. This shows a sky frame formed during the original processing of one of the September nights. Unfortunately, the random component is also about the same level (~1% of sky) and even after sky subtraction affects some science product frames. After some investigation it was clear that most of the created September flats suffered from this effect as well (contributing to the problem when used in processing) and after a series of tests to ensure that other flat field characteristics had not changed during the camera engineering work we decided to use the August flats for processing. So far we have found that the pipeline-created sky frames are a good diagnostic of the problem being present but are still investigating which nights are affected. To date we have not seen this problem in master dark frames produced by the pipeline. With some misgivings about its general use, we have developed a prototype sledge hammer software fix which if all else fails we will use on affected data in an attempt to mitigate the problem. Unfortunately, this has to be run at the individual frame level, ie. post-sky subtraction but pre- interleaving and stacking, due to the random nature of some of the effect. This will involve selective partial reprocessing of affected MBSs and possibly significantly add to the overheads of running the pipeline. Reprocessing: In total 17 nights of UDS data (~700 Gbyte of products) for 06B has had to be reprocessed to allow for more rapid sky tracking to remove some odd features in the science product frames. The reprocessed data has been supplied direct to the UDS consortium for them to finalise the deep stacking and thence make the deep stacks available for DR3. Photometric calibration: STH has circulated a first "real" draft of the photometric calibration paper to MJI, PCH, and SJW. He asked for group members to have a read and send in comments (and presumably anyone else reading this). The paper can be found at http://www.ast.cam.ac.uk/~sth/photom_draft.pdf. In the meeting, discussion focussed on the interpretation and possible causes of the spatial systematics. STH also asked for thoughts about how to better demonstrate the effects of non-photometric conditions on the quality of the calibration. He showed a plot of sigma_ZP (taken as the MAD of the MAGZPT values across all 4 chips) vs ZP. DWE suggested investigating repeat photometry in the different regimes of the figure. STH will investigate if he can use the per chip scatter in ZP instead of the inter-chip scatter as a more 'informative' measure of non-photometricity. MJI hopes not because he doesn't want to update all the headers yet again. STH hopes to circulate the paper to a wider audience for comments in a couple of weeks. <<<< MJI noted that due to the new calibration protocol involving individual detector photometric zero-points, that for a trial period data would be flagged as ok_to_copy in monthly batches since it requires a month of data to accurately compute the corrections. The advantage is that this saves having to keep track off and to keep updating the photometric zero-points, the disadvantage is that it slightly delays release of flat files through WFAU. MR noted that the WFCAM quality control database has been updated to include the latest version of the photometric calibration and some of the recently reprocessed nights (but not UDS). 7. VISTA news Pipline Progress: A number of changes were required to the DRL to make it CPL 4.0 compatible (the version which uses cfitsio, not qfits). This was fairly urgently requested by Sandra Castro because ESO want to install this version in the Paranal pipeline. v0.7 of the software and v1.9 of the DRLD has been released. Problems with using FITS compression via CPL have been identified and reported. There was also a new version of the 2MASS catalogue released with this. Camera Test data: The July Paranal engineering run produced 2,500 files which have been used for various tests (of the system for VPO use and for DRL development); reports were issued and discussed by telecon. It was agreed that it would be unwise to offer READ-RESET mode observing initially. Naidu circulated an interesting conversion gain non-linearity paper prior to the telecon which highlights some interesting features of NIR detectors including VIRGO and Hawaii-I (the paper is available on the internal web pages - mind you it still leaves several open questions related to non-linearity). Tiles: Invesigation of the behaviour of the VIRCAM/VISTA software when doing tile offsets showed up a number of problems: namely in the computation of the WCS rotation matrix; a sign error; and a (just) significant rounding error in the propagation of the sky position angle. It is fully expected that first-light will demonstrate further geometrical transpositions et autre faux pas. Survey PIs and Science verification: ESO have reminded survey PIs that the science verification program will not simply be a case of performing part of the surveys, but will be an independently constructed program. In light of this MJI should raise the issue of science verification of VISTA data at the next VDMT meeting. CASU would like to be in the loop on this if possible. <<<< Letter to PIs: No further enhancements have been proposed so we will do the enhancements already planned. Some issues have been flagged in the present system which we will need to consider for VISTA: a.) the survey overview pages are not in sync with the processing pages, ie. the user goes to the processing pages and see that the data is processed but that is not included in the survey progress pages. This is due to the database design; b.) UDS frames (ie. with UDS in the project ID) have been observed that are not part of UDS so some statistics were wrong for this survey. Corrected now. MJI noted that Henry McCracken at Terapix wanted access to a full VISTA header set in preparation for dealing with the Ultra-VISTA data. He was sent a full simulated raw MEF file. 8. ADASS 2007 ADASS XVII proceedings: About 150 contributions have been received with 80 still missing. The deadline passed on November 2nd. Meanwhile RWA will check all the contributions received so far and will delete extraneous files and rename those papers with the wrong ID - during this time it will be possible to include late arrivals. A date of the end of normal working in December (ie. ~20th) was set for the final compilation of the latex source prior to shipping to the publishers. It was felt important to have the proceedings published in time for the next ADASS announcement in April. 9. AOB DWE said that apm1 was possibly not the best place for the Sky Server of the Wide Field Survey object catalogue. Currently it is causing fragmentation problems. Should we get a larger machine or move the location of this catalogue to a more maintainable system? It was decided that the catalogue should be served the same way as IPHAS is. Once this is done, Guy Rixon should be informed so that the AstroGrid registry can be changed. EGS was volunteered to email JMI to get the necessary information so that this work can be done. <<<< Scheduled updates. - apm25 and apm26 patched with latest updates and rebooted. The remaining APM machines will be rebooted during quiet spells (ho hum!). There was some further discussion about security of the APM cluster. MJI will review the user accounts and for now lock any unused ones prior to removing them. <<<< Membership of the new IoA Computing Advisory Committee was noted. Continuing Actions ------------------ none ! New Actions ----------- MJI forward access details of JAC wiki page for WFCAM observing summaries MR advertize illumination correction information PSB acquire wireless for APM building (and get order book at same time) MR perform backup verification test using 4 old LTO tapes MJI make latest reports to UKIRT board available on internal pages RWA skip old data tapes MR email ESO regarding raw data transfer interruptions ALL provide comments on photometric calibration paper MJI raise issue of science verification of VISTA data at next VDMT meeting EGS email JMI regarding details of database on apm1 MJI review user accounts on APM cluster