2nd May 2003 Present : MJI, JRL, PSB, DWE Apologies: STH Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes of 030425 3. Implications of possible new readout mode 4. Possible use of various compression algorithms 5. Observing strategy meeting planned in June 12/13 at ATC 6. What's been happening about the sky flat data that AA took 7. Report on Ultrium tape drives 8. Progress on deliverables in WPs 9. AOB Minutes ------- 1. Actions from last meeting STH has looked at CDR document 7 (photometric calibration) and is liaising with NCH - ongoing need to think about and discuss calibration plan at next meeting <<<< DWE has read Ian Bond's MN paper and contacted him re. difference imaging. DWE will obtain the software and test it out. <<<< MJI mosaicing requirements: continuing action <<<< JRL has investigated image compression (see item 4 notes on web page) JRL naming of the master calibration files: continuing action <<<< DWE has contacted Steve Warren re. UKIDSS Survey responsibilities and planning (see item 8) MTB has investigated Ultrium LTO-2 tape drives (see item 7) JRL observer comment database will be mirrored at Cambridge with the comments being placed into the FITS headers at the raw MEF stage. ALL reported progress on work packages prior to meeting **** STH has started thinking about naming conventions for documents and has had some feedback from JPE. This will be discussed at the next meeting. <<<< DWE has altered links to diary page to take you to the bottom ALL organization of diary page: the new entry point seems a satisfactory solution. PSB has collect some of the information needed for VDFS documents for PDR but still needs more input on the science/user requirements. He will contact Will Sutherland to progress this (see item 8.) **** MJI suggested that the detailed reporting regarding the work packages should only be done once a month prior to the monthly management meeting unless there are urgent issues that require more immediate attention. A reminder will be sent our prior to the relevant meeting. 2. Comments on WFAU minutes of 030425 There were none (comments that is). 3. Implications of possible new readout mode MJI explained that with the proposed new readout mode the basic exposure will be 5s, immediately followed by a small shift (<~10 arcsec) and then another 5s exposure. This will be used to remove most of the "bad" pixels so has no impact on the science archive data volume (providing the shift is a constant offset => no increase in no. of confidence maps required). DWE asked about how much of an overhead this will cause. It is thought about 10%. The biggest difference with this mode will be that the data rate coming out of the DAS computers will be 2 times larger than anticipated. MJI has recomputed the expected data rate. Allowing for an average of 20s dead-time in every minute (which is possibly generous) WFCAM will generate 300 Gb of raw data in a 10hr night [r*4 x4x2kx2k x8 x60 x10]. It was felt that as it stands this is too close to the margins for cpu, i/o and data transfer thro the summit pipeline. JRL and MJI propose to convert the DAS output to 16 bit FITS (which it effectively is anyway for a 5s exposure) and then consider using lossless Rice compressed FITS as the i/o transfer medium. The total reduction in i/o thro the system would be around factor of 4 - as would the amount of data requiring dumping to tape. This also gives a similar reduction in i/o at ingestion of the data into the Cambridge pipeline. 4. Possible use of various compression algorithms JRL reported on the tests he had done on various compression algorithms. The most promising is the Rice tile algorithm built into cfitsio, which is specifically designed to work with images. Rice is much faster (~x10) than gzip and bzip2 and also gives a better compression factor. JRL explained how tile compression worked and noted that the (specifiable) size of the tile was not critical regarding compression factor. PSB was concerned if any of these algorithms were lossy. In particular, there might be subtle effects present in stacking lossy data. For integer data, Rice is lossless. The notional R*4 data from the telescope will be converted to U*2 with no loss of information with the current set up (factor 2). Rice compression will give another factor of ~2 with no loss = total factor ~4. For intrinsically real data (which is a loss already for integral data) the amount of loss can be specificied eg. 1 bit, 1/2 bit 1/4 bit and so on (JRL's report for more details). STH/MJI suggested that we do some tests with lossy compression of real data to see if it has any noticeable effect on image parameter estimation. This can be done on some Wide Field Survey data. Another test to carry out is to see if the losses are random, since this may be important for stacking <<<< DWE suggested that the data be degraded to simulate undersampling to see if this has implications on parameter estimation after compression. Make it so. <<<< Compressed data could also be used to alleviate transfer to WFAU. (Aside comment: JKT ING online raw U*2 archive data on RAID should be compressed to save space - disk space will ALWAYS be at a premium) 5. Observing strategy meeting planned in June 12/13 at ATC Is this an observing stratgey meeting or a general ukidss meeting ? It was felt that possibly too many other topics were on the proposed agenda (eg. do we really want another Science Archive session and another Pipeline session ? - the documents are all on the web). The main purpose of the meeting should be the observing strategy ie. planning the surveys, monitoring their progress, the survey definition tool, designing processable MSBs for the main surveys, planning the commissioning strategy, calibration strategy, observing schedule and so on. All read, think about and send comments to MJI on Mark's proposed agenda. MJI will then collate and contact Mark. <<<< 6. What's been happening about the sky flat data that AA took ? Some J-band twilight sky flats have been taken. JRL has pulled these across. Tests on these will follow. <<<< [meetings do have their uses then] 7. Report on Ultrium tape drives MJI said that the native capacity of the LTO-2 drives is 200Gb (ie. uncompressed) and that the data transfer rate is supposed to be 100Gb/hour. There is a clear upgrade path -> LTO-3 (300 Gb) -> LTO-4 (400Gb). The drives can be rack mounted and come with fibre channel or fast scsi connectivity. Tapes can be grouped by cartridge (8, 12 .....) and then form the basis of a multi-Tbyte library system. The drives appear to be about $6000 each (probably £6000 here). HP, IBM and Seagate signed up for this protocol which implies a standard and a non-proprietary format. WFAU are proposing to use this type of system for backups so the 3 sites having the same type of mass storage tape system gives us a fallback position for data transfer from Cambridge to Edinburgh. We are considering buying 2 drives (~£12k) possibly within a library system (useful for backups of other RAID arrays). MJI to contact WFAU and see if we could get a competitive quote through their supplier. <<<< [Who is supplying the 10 drives for the summit computing ?] PSB was asked to contact our usual supplier regarding these drives and get some quotes. <<<< 8. Progress on deliverables in WPs All the people present gave a brief presentation of progress with the work packages they are responsible for. STH sent in his notes by email. PSB 1.1 PSB has been working on the VDFS PDR documents but could not find 1.2 all the information he needed on the Vista web pages particularly on 1.3 the science/user requirements side. MJI dug out the relevant most recent documents he had (after the meeting), namely vista_srd_v20 and vista_ocdd10, and put them on the internal web page before the action became official. PSB is using Word and drafting out the section/subsection structure for later fill-in. It was clear that revisiting the Vista science drivers would be useful (they are a bit out of date) to check if the current science requirements are adequate/appropriate in the light of changes since the science case was assembled. Pete to contact Will about this. <<<< STH 2. STH + MJI produced spreadsheets of the costs of the ESO deliverables and keep updating and maintaining the WP spreadsheets, gantt charts 2.5 define Observing Protocols: ongoing, progress with JRL and Paul Hirst on tile keyword propagation. 11.1 photometric standards and calibration: STH is writing general 11.2 document on photometric calibration to take up where NCH's document left off. Has contacted NCH to sort out handover of this task and should present at June workshop. <<<< Has obtained and is now examining 2MASS standard stars from 32 selected regions. Assessment of their suitability in progress. Also investigating choice of several master calibration fields for WFCAM using Globular Clusters. Has discussed with Paul Hirst possibility of observing these with UFTI/UIST. MJI 2.1 ongoing and set up to do via the fortnightly meetings, minutes and monthly reports 2.2 discussed and agreed baseline transfer mechanism of data from JAC to CASU; and outline hardware setup for data acquisition and summit pipeline 12.3 has arranged to acquire a copy of the 2MASS data products. This is about 80Gb and will be supplied asap. 13.1 devised a procedure for creating master catalogues for multi-band data and incorporated it in pipeline toolkit development software 14.1 bug fixed and enhanced current version 1.0 stacking, and mosaicing software. Propagating changes into release (ie. C) version of code. Dumped current (old) version already distributed to WFAU into CVS. 15. contacted Pierre Astier from the SNe Cosmologoy Project about acquiring the SNe adaptive kernel matching code. This is in parallel with DWE's work with Ian Bond's code. JRL 2.8 solved connectivity problem with JAC's archive server. Working towards mirroring of comments and progress database. discussion with JAC and ATC staff regarding: 2.4 (a) grouping of observations into contiguous tiles 7. (b) methods to aid in synchronisation of pipeline modules between CASU and JAC 7.8 evaluating image compression schemes to speed up data flow. 7.1 CVS software modules --> CVS repository at JAC 8.1 ongoing investigation of optimal ways of removing instrumental signature with flats from various sources (sky, dome, et.c) and sky subtraction. 8.1 devised, wrote and tested general purpose pipeline module for correcting crosstalk. 8.1 ongoing work on general purpose pipeline module for persistence. DWE 2.7 UKIDSS survey planning: altered UKIDSS survey coverage plotting programme so that user-defined areas can be added. Equivalent 2-year plan plotting programme has been written. Coverage of Data Release 1 for SDSS has been incorporated into these programmes. These plots will be placed on the web and sent to SJW. <<<< 2.6 UKIDSS survey planning/observing strategy: discussed with SJW the 2.7 responsibilities division between UKIDSS Survey Scientist and Manager ongoing, find out more. <<<< 2.9 system documentation: updated web pages 15. difference imaging: read Ian Bond's MN paper and contacted him regarding possibility of using his software for tests on image subtraction, ongoing. <<<< 9. AOB - miscellaneous MJI said that JPE had a meeting at ESO with Peter Quinn et al. regarding transfer of raw WFCAM data. Unsurprisingly ESO do not want the data delivered on tape. Transfer over the Internet is a possibility that is being further investigated. JPE had also forwarded information about a lot of relevant ESO documents on their Common Pipeline Library (CPL), contact details and a new version of their Data Interface Control Document plus assorted FITS header information. MJI noted that we already have processed raw ESO WFI data and can get ISAAC data out of the ESO archive as needed to inspect the FITS headers directly. JRL ongoing evaluation of Sybase IQ RDMS for possible use with WFCAM and other VO catalogues with a view towards the demands that will be made by VISTA impressive performance, but a price of £30k!! per CPU? was mentioned. PSB said that the internal directory was looking a bit crowded. DWE said he would look into this and do some reorganizing. <<<< MJI noted that our part time software engineer had produced prototype standalone software for generating colour images (PNG) from 2-3 passbands, automatically using the WCS information for coaligning and rescaling using robust level estimators. This is useful both as a visualization and a publicity tool. In response to a sneaky question from JRL, MJI noted that there was no further news on the CASU grant, it has clearly entered a stasis field. Paul Hirst from JAC was visiting from Mon-Thur which lead to much useful discussion on various WFCAM topics - thanks Paul. Finally, MJI estimated that short term management overheads due to meetings, minutes, reports have gone up by 0.2 FTE since 1st April. Need to revise upward management overheads in WP spreadsheets. <<<< ============================================================================== Continuing actions ------------------ STH photometric calibration plan and liaising with NCH DWE obtain Ian Bond's difference imaging software MJI find out what the mosaicing requirements really are and who is doing what JRL more deep thought about the naming of the master calibration files. STH naming conventions for documents: report conclusions. New actions ------------ PSB contact Will Sutherland for help with the science case and science/user requirements aspects of the VDFS PDR documentation. MJI test effect of lossy Rice compression on parameter estimation. Are the JRL effects random and non-systematic? Does undersampling have an effect? STH ALL send comments re. Observing Strategy meeting agenda to MJI who will then contact Mark Casali. JRL carry out tests on J-band twilight sky flats. PSB contact local supplier re. Ultrium tape systems MJI contact NCH and see if we can get a quote from their supplier STH present photometric calibration plan document at June meeting MJI Acquire 2MASS data. DWE Place DR1/UKIDSS plots on web and send to SJW. DWE contact SJW re. survey planning/responsibilities and if possible attend meeting between SJW and AA DWE reorganize (simplify) internal directory structure STH revise WP spreadsheets to reflect increased management overheads MJI