10 February 2004 Observatory meeting room 10.00 - 12.30 Present: MJI, JRL, PSB, DWE, STH Apologies: WJS, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report on JAC telecon of 28th January 4. Report from PPRP meeting and VDMT of the 4th February 5. Status of ESO PDR deliverables 6. Data transfers to WFAU and Garching 7. Status of ISAAC reductions 8. AOB Minutes ------- 1. Actions from last meeting STH met with Paul Hewett at the end of last week to discuss various photometric calibration issues that concern WFCAM and VISTA. The work has been divided up between them and should be done by the end of next week - ongoing <<<< STH checking the spatial systematics in the UFTI data is an ongoing task. No more UFTI data on this issue is foreseen, so this should be completed by the next meeting. STH also added that he has been in touch with people at Caltech regarding CSO Tau data to see what (if any) correlation exists between this and extinction, as part of the photometry work package <<<< JRL has sent some dummy WFCAM FITS headers to Benoit and Adam. This probably will need one more iteration. The clarifications needed could not be obtained from the example SDF header official WFCAM document. <<<< MJI has been in touch with Andy Adamson and he (AA) will ask the ATC for the requested example SDF file(s) with all header packets present. This is required for a full end-to-end test. JRL have looked at the issues involved in the CPL/QFITS route (see item 5) PSB MJI asked AA at the JAC telecon about the progress of the engineering tests. Apparently coplanarity tests have just been completed, optical alignment comes next. A further email from Mark Casali said that they had cooled 2 science grade chips down and planned to cool the other two down shortly before commencing more coplanarity checks followed by device testing. MJI has been in touch with MMC and confirmed that the data from the lab tests do not need to have the full header packets in them and that we would prefer them in FITS format. Transfer mechanisms to Cambridge are still under disucssion. STH finished the various GANTT charts and other documents needed for the PPRP and VDMT meetings. JRL produced copies of his UFTI reports for the PPRP meeting JRL JRL passed on the details of the missing ISAAC frames. Although the STH files were present, they only contained the headers. STH is still in the process of tracking these ~100 or so frames down - ongoing <<<< MJI has carried out the stacking of the ISAAC data (see item 7) STH has acquired the FIRES reduced data to compare with ours. MJI said that this example of the FIRES data had probably been stacked to maximise faint image detection (ie. using some sort of matched detection filter) hence the noise is smoothed (ie. correlated) over several pixels making direct comparison not so straightforward. Visually the two sets are comparable. He will do some further catalogue-based tests <<<< STH calibrating the FIRES data is an ongoing task <<<< JRL has written an outline of a report on the ISAAC reductions. This will be iterated with MJI - ongoing <<<< JRL has done tests on the ISAAC data doing flatfielding first, followed by sky estimation and then sky subtraction. The result was very little difference, a 0.3% to 0.5% improvement in sky noise. The motivation for doing the processing this way is to improve the rejection of faint images (and faint extended halos) in the sky estimation phase. MJI WJS has been invited to these meetings (but is in Edinburgh this week). ALL sent in end-of-month reports to STH STH emailed our VDUC suggestions to JPE. Tim Naylor is to be the chairman, but the rest of the committee hasn't been finalized (still waiting for some nominations to come in). It was thought unlikely that a VDUC meeting would be held before March. One of the first issues CASU would like VDUC to consider is the pipeline (and archive) User Requirements for VDFS. WJS is drafting a first version of this. MJI have discussed dome flats for VISTA with WJS and confirmed that 10-20% PSB variation in illumination across the array was fine. The main requirement (for non-linearity measures) is to have stable illumination. 2. Comments on WFAU minutes MJI noted that WFAU had asked us to consider checksum tests for the data transfer, together with a number of related issues in an email - see item 6. Nobody could figure out what "mouse-bat-folicle-goose-creature-ampersand" referred to. 3. Report on JAC telecon of 28th January A number of questions were put to AA: 1. engineering chip tests: no news at the time but see earlier item 2. getting SDF file for end-to-end test: AA will see to this 3. dome flats: confirmed that we don't expect super uniformity 4. more UFTI service data: confirmed none is planned and agreed none wanted 5. commissioning personnel: agreed to outline CASU plan but noted CASU team probably not required on-island before mid-August 6. CSV AA & AL: found out about progress (confidential) MJI reported that AA wanted to know more about the various reviews coming up this year and was concerned (as is MJI) about the number of them and their possible interference with WFCAM commissioning. The next telecon is scheduled for 18th Feb. This is delayed by about a week due to JAC being busy with Michelle recommissioning. 4. Report from PPRP meeting and VDMT on the 4th February MJI reported that the PPRP meeting, a full house with standing room only, seemed to go well and that we should hear officially shortly. There was a lot of detailed questioning on several aspects of the programme. MJI noted that having a uniform presentation style for the 4 talks (JPE,MJI,NCH,GIL) helped to create a better impression, thanks to ???? for suggesting it and to NAW for organising it. WFCAM and VISTA operations was deliberately not included in this bid and will have to be bid for later. STH suggested that we revisit the notes taken from DWE's and his meeting in Leicester in order to get some further ideas regarding the running of operations. The VDMT meeting was an anticlimax by comparison. The main actions of relevance are that: someone from CASU should attend the VoProc meeting in Leicester on 25th February (JRL said that he would go since the topics were more in his domain of expertise); the EIS MVM pipeline is available and MJI was asked to report on its relevance, or otherwise, to VDFS. <<<< 5. Status of ESO PDR deliverables and QFITS -v- CFITSIO JRL reported on his comparison of the functionality of QFITS compared with CFITSIO. A start on this was made by going through the pipeline software modules and noting what additional functionality would be required in QFITS if the pipeline code had to be converted from CFITSIO. The top-level overview list is: (1) lack of Rice tile compression (2) the ability to read a subset of columns or image pixels (3) the ability to access single extensions and image subsets within a file using the following notation: fff.fits[1] fff.fits[1][100:150,200:237] (NB: This is not a FITS standard, but it seems to be common practice in many FITS packages: CFITSIO, FUNTOOLS, IRAF, etc) (4) some, but not all, routines return an error status and there is no inherited status in the sense that subsequent calls ignore the current status (5) lack of header keyword 'classes' -- allows you to exclude certain classes of keywords when building a new file from an old one. (6) a good user's guide. The reference manual is not particularly detailed and too much is left unexplained. (7) no Perl interface (there is a restricted Python interface) (8) no WCS interface PSB pointed out that although these features are not be in the current version of QFITS, it is possible that perhaps some of it will be in the next version. It was agreed that the way forward here was to make a detailed list of the functionality required if CASU were to adopt CPL+QFITS and to include it as requirements in the Data Reduction Specification Document. PSB also noted that CPL might have some of the required functionality. He and JRL will continue the comparison. <<<< PSB at this point showed some plots of a simulated VISTA file that he had generated and compared the DS9 browser (that seamlessly mosaics on-the-fly given a WCS) with the RTD screenshots that SMB had obtained from the simulated data. SMB had determined from his tests that while the proposed VISTA data format was fully compatible with RTD, that RTD currently has a couple of shortcomings including not being able to zoom out far enough to display the whole mosaic in place. 6. Data transfers to WFAU and Garching MJI reported on email exchanges he has been having with Eckhard Sutorius. WFAU want some form of external md5 checksum carried out on the processed WFCAM files before they are transfered. All agreed that this seems reasonable, even though standard TCP/IP transfer protocol does something similar anyway. There are various trivial alternative ways of implementing this, "openssl md5 filename" (also "md5sum filename") and built-in routines for doing this also exist in standard Perl distributions. MJI noted that ESO have asked for something similar for raw data transfer to Garching, but they wanted to provide their own checksum routine for doing this. DWE noted that an additional use of an external checksum would be to verify that files haven't been modified since the checksum had been generated. This seemed like a good additional reason and will be implemented. Other matters discussed were: * There will be one directory per observing night. This is the way it will be processed and structured ie. on a nightly basis. * The issue of handling read-readiness will be done via a lock file set in the directory while the data products are accumulating in it and after the processing is complete the lock file will be removed. An alternate strategy would be to have a go file created when the data is ready to process. Another strategy suggested was to process the data in a temporary directory and rename it when the data is ready. Any of these would do. For now we will stick with the lock file and revisit the issue when we actually get vast amounts of WFCAM data processed. Linked to this was the request from CASU that a receipt file be created after WFAU had copied and verified the data. * The expectation is that reprocessing will occur very infrequently and (probably) in large batches (eg. after bug fixes). This will require CASU to inform WFAU of the fact that reprocessing is happening and why. The data will go into the same directories. The file headers will contain all the versioning information. * no processed data will be officially released until CASU are satisfied with it. The exception will be test data for ingestion experiments. JRL reported that ESO have asked for some more extra keywords to be added eg. explicit RA and DEC keywords with keyword values in decimal degrees, an AIRMASS keyword etc...... These will be inserted on-the-fly during data transfer. 7. Status of ISAAC reductions JRL has finished pipeline processing the OBs, bar the few missing ones. MJI showed some images of raw data (6x20s coadd); the same dark-corrected reset-anomalied, and flat-fielded (limiting mag ~20); and the output of a single OB (40 such K-band frames suitably dithered and stacked, limiting mag ~22). His further stacking experiments using the ISAAC data revealed a number of interesting low level remaining instrumental artefacts, that only become apparent on deep stacking. These artefacts were removed using extant parts of the pipeline toolkit - at the expense of any real large scale features. It was noted that the FIRES team had done a lot more data massaging before stacking cf. Labbe et al. This data covers the Hubble Deep Field South and is a ~3x3 arcminute square. The stack of all 44 K-band OBs (representing 60 hours of integration on an 8m telescope) has a limiting magnitude fainter than 24th in K. A pseudo-colour image using all the stacked J H K data was also shown. Visually it is difficult to compare this reduction with the FIRES project reduction since their version has been resampled onto a finer grid and also smoothed. MJI will do further catalogue tests on the two datasets. The data will be made available to the Subaru group in order for them to perform further stacking tests. This work will be incorporated in the report actioned earlier. 8. AOB --- MJI said that there would be another APM building refurbishment planning meeting tomorrow, which will be reported on at the next CASU meeting <<<< DWE showed the latest iteration of the documentation web page with the document numbers on. This will seamlessly become the official documentation page by the end of the month. The only request was to reduce the overall lateral size by shrinking the description column slightly <<<< If there are any more documents that need numbers allocating to them DWE needs to know about them. <<<< DWE will then send out an email to the authors of the documents that need a number inserted. <<<< WJS's draft of a draft VDFS User Requirements Document was discussed. It was noted that this was the UK version and not an ESO requirements version. A number of issues were raised but it was decided not to discuss this in more detail there and then but rather arrange a meeting with WJS for next week to go through it page by page. [A slight change of plan from that discussed at the meeting.] <<<< Continuing Actions ------------------ STH finish the bright star checking of the photometric standard fields (by end of next week) STH look at the spatial systematics in the UFTI standards data and new test data (by next meeting) JRL iterate on WFCAM example FITS headers and files with Adam and Benoit STH acquire missing ISAAC data MJI compare FIRES project reduced data with ours STH calibrate FIRES data using NICMOS or the published FIRES work JRL write a report on ISAAC reductions MJI New Actions ----------- JRL add in CPL functionality to QFITS -v- CFITSIO comparison PSB JRL attend and report back from VoProc meeting in Leicester MJI attend APM building meeting (Wednesday) and relate saga next time PSB DWE alter documentation web page (reduce the overall size and shrink the description column) ALL inform DWE of any documents that require document numbers DWE email document authors of the numbers that should be inserted. ALL read and make notes, if necessary, draft VDFS User Requirements Document in readiness for a meeting with WJS next week