4th February 2009 APM meeting tables 11:30-13:00 Present: MJI, JRL, STH, MR, EGS, RGM, RWA, NAW, SC Apologies: DWE Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- 1. Actions from last meeting STH the technical pages are still not up to date and other inconsistencies have been pointed out in terms of current WFCAM processing practice jarring with existing information, MJI will help with the updates <<<< JRL has not produced a web page of HawkI publicity for CASU, deferred due to other factors but still - ongoing <<<< MJI have not finished off the stage-I software release pages - ongoing <<<< EGS STH has put the WFCAM calibration paper on astro-ph, but RGM pointed out that the astro-ph version of the photometry paper has a couple of messed up figures. STH will fix this and put a high resolution version of the paper on our own website(s). - ongoing <<<< MJI has not finished off the technical paper - ongoing <<<< PSB compared Calibration Plan with current expectation for calibration and found it needed updating (see later item) EGS made sure the CASU presentation from the ESO Public Surveys meeting in September was available on the ESO website. MJI pre-empted expected criticism re: deblended Petrosian fluxes and publicised the fix STH contacted Mark McCaughrean about HAWKI Eta Carina data status MJI forgot to contact Mark Casali about HawkI radial distortion - ongoing <<<< RGM liaise with JRL about processing HAWKI PI data - still ongoing <<<< JRL contact Brad at JAC regarding the Nexp > 1 issue - happened and is now solved STH further investigate photometric features in available VISTA data (throughput, colour equations, sky brightness) - deferred due to catalogue bug affecting VISTA catalogues as well, they need regenerating prior to doing this. Dropped from actions since part of longer term WP. JRL sorted out project names of VISTA MSBs taken by STH as well as they STH could 2. Comments on WFAU minutes CASU would like to thank Mike Read for spotting the faux pas with the catalogues. Next time we meet up the drinks are on us. MJI noted the comments on the UKlight ups and downs and crushings and pointed out the resubmission of the bid and the new contact names involved. He also noted the hardware server moves to the new room over last w/e and wondered if the ensuing silence was good news or bad news. RGM raised an issue about version control and the current difficulty of knowing which files were processed with what version of the software. There was a fair bit of discussion on the best way to improve things since the current versioning relies on FITS header items from each software component used in the processing and for bulk reprocessing relies on a global version flag set in the datesubdir. There are two main problems with the current strategy: the CVS id as currently implemented fails to pick up lower level changes in embedded subroutines resulting in header processing id's that do not reflect ALL the changes; the lack of a simple overall version flag in each processed product file i.e. without a bit of detective work if only a few files in a datesubdir have been reprocessed its not completely straightforward to recover the event sequence. JRL has been figuring out a way to get around the id problem and this should point to a solution for the version issue too. <<<< 3. Recent and upcoming meetings EGS reported on a recent EGAPS meeting at the IAC, Tenerife 22-23 Jan 2009. Every IPHAS field has been observed at least once now. Fields with very poor data quality will be included for re-observation in the hopefully final proposal to be submitted for period 09B. Regarding the data release the global calibration of this GP surveys is not going to happen in the near future due to delays in recruiting, so it was decided to extend the IDR with all the currently available data. Better statistics regarding who is using it are required and this will be done via suitable tagging when the data is downloaded. The consortium also want to improve the visibility of IPHAS in the astronomical community by contacting Vizier and asking them to make the IDR and subsequent releases available from their web interface. There was some discussion about the best way to proceed here given the IPHAS IDR is already published through the VO. From the UVEX perspective they also asked to have data access similar to the one provided by the IPHAS IDR and preferably with the IPHAS and UVEX data already cross-matched. MJI and EGS had already discussed the best way to do this and decided to do the cross-matching when the band-merged catalogues are produced. MJI noted that a UKIRT board meeting had taken place since our last get together and thanked the Board for pressing STFC to progress the then stalled CASU and WFAU rolling grant issue. MJI, STH and RGM attended the UKIDSS RAS meeting held just before Christmas. All the presentations from the meeting are available at http://www.ukidss.org/rasworkshop08/talks. PLATO - STH+NAW attended the last PLATO consortium meeting at ESTEC (Nov. 28th 2008), and will be attending the next Data Centre meeting to be held in Paris on Feb 12-13. The IoA were awarded some modest funds by STFC to enable us to continue investigating the potential of the mission and our role within it. NAW pointed out that there is a Euro VO workshop March 30-April 2,2009 in Garching (http://www.euro-vo.org/aidahandson2009/registration.html) Deadline for registration is February 15, 2009 if anyone is interested in going. 4. Data archives update - AAT, ING, WFCAM, UKIRT An easy one, all archives are up to date, more or less. MJI asked about recent backups for the raw data archives. AAT is ok, as are UKIRT, WFCAM and ING. The latter two are received over the internet and a hard copy is made to tape and stored in suitable chunks after they have been ingested and verified. We will adopt a similar strategy for VISTA. MJI asked EGS to separately backup the ING 2008 data since although the raw transferred files (*.gz) are backed up to tape monthly they are reformatted to Rice-compressed FITS and assorted illegal FITS headers fixed before archiving them on-line. <<<< The WFCAM'ers noted their surprise at getting an email from PSB on January 30th, and subsequent days, but it turned out to be from his cron job running on apm6 collecting WFCAM raw data. MR has already been in contact with JAC about taking over the responsibility for the transfer scripts and while that is being set up in parallel we will continue with PSB prompting us. 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, ESO WFI Optical processing: EGS continues processing the ESIS data with the usual customers for the data (VISTA/VIDEO, Herschel/Hermes, Spitzer/Servs). HAWKI: JRL is regenerating the catalogues in light of the catalogue bug though we note that image nebulosity filtered versions of the catalogues (already shared with the Carina team) are free from de'bug since they were generated with the toolkit catalogue software. STH is investigating some possible issues with the spatial distribution for colour-selected sources raised by the Carina team. JRL will also reprocess the narrowband data to try and improve the sky subtraction; the broad-band <<<< data is as good as its going to get. Meanwhile MJI will try and remember what he was going to do next for this project. All the new panoramic M31/M33 survey data from MegaCam for 08B arrived and have been processed, stacked catalogued and glued together and the results are awesome. The usual INT WFC camera processing for UVEX, IPHAS and the odd PI prog has been done. 6. WFCAM update WFCAM news: its back on and taking data. Version 3 (sic!) of the flat screen will become available this semester and we have suggested taking some linearity sequences to check how WFCAM fares. Data transfers of 09A have commenced, the nexp > 1 scaling problem has been fixed closer to source, the OMP DB is communicating and the first few nights are sitting ready to go ...... well almost since we are still eagerly awaiting some sky flatfield data so we can begin. MJI moaned that he could find no trace of the updated compress routine in the header, but it turned out he was looking in the wrong place and it is there in the SHDUs - the PHU one being misleading. As he was in rant mode he also whinged about the fact that ds9 still bombs out when uncompressing the raw images even though there is nothing wrong with them. JRL opined it was to do with some ancient cfitsio version being used at JAC. He will check and help sort this out since it is a an operational pain having to uncompress raw files first before being able to examine them in ds9. <<<< Bug news: a rather subtle bug to the catalogue generator was introduced on 15th April when some very minor fix to deal better with heavily saturated stars (sic!) was introduced. Unfortunately due to its unpredictable (and hard to spot) consequences this has required us to regenerate all the catalogues from 20080312 onward (i.e. the rest of 08A and all of 08B) and has lead to some delay in the release of DR5. It has also required recalibration of the photometry, detector zero-point offsets and illumination corrections, though the only changes to the images have been updated photometry coefficients in the headers (typically only at the 1-2% level). Generally all the smaller aperture fluxes were ok hence the overall photometry was only marginally off contributing to why this wasn't picked up during our normal checking procedure. The bug was found and fixed in short order and some good news in this saga is that the VISTA processing kit regenerated all the required 08A and 08B catalogues far faster than we expected. All the reprocessed data has now been checked off and awaits retransfer to WFAU. How to avoid this happening again (or at least minimise the chances) was discussed in detail offline prior to the meeting, though at least one of the issues raised there, better version control in the headers, is commented on in 2. above. As part of the bug fix we also implemented, as an extended test for the 08B recataloguing, the mod to improve the Petrosian, Kron and Hall (PKH) radii and fluxes for deblended objects. The other piece of good news is that this mod can be back-propagated to all the earlier catalogues from 05A ->, i.e. these parameters for deblended objects can be fixed in place without having to regenerate the catalogues. Reprocessing news: the UDS K-band reprocessing from 05B was completed over the Christmas period and made available to the UDS folk for assessment on 2nd Jan 2009 - is no news good news ? The PKH flux/radii fixes for 05a-08a inclusive are ready to go as soon as the recatalogued 08A data has been copied. The Orion nebula data for U/05a/100 (taken in 05B and 06B for Catarina Alves de Oliveira and Mark Casali) was reprocessed from scratch to improve the sky subtraction. This data also provided a good independent test of the nebulosity filter developed for GPS photometry in regions of high nebulosity. We await results with bated breath. Ricky Smart contacted us at the UKIDSS meeting about the parallax programme. Although satisifed with the results he was curious that his own detailed attempt at PSF-based astrometry actually gave worse results than the pipeline astrometry. MJI suggested we try "dribbling" the 2x2 interleaved stacked data to improve the PSF characterstics and provided him with the appropriately dribbled frames. Although the PSFs were now well-behaved (i.e. not "spiky") RS's PSF fitting still gave results not as good as the pipeline astrometry (which just shows you how damn fine it is and how well the sneaky ML shortcut we implemented several semesters back works). This still leaves us with the extant GPS reprocessing in nebulous regions to improve both detection and photometry, stalled due to the recataloguing of 08A and 08B noted earleir. Other news: the photometric calibration paper has been accepted, posted on astro-ph (figure faux pas not-withstanding) and awaits proofing. MR noted that one intrepid user (Fraser Clarke in Oxford) had asked for a copy of the processed UKIDSS/B2 data he had taken during 08B since he wanted to select by eye the best frames to stack and stack them himself. Aforesaid individual was presented with the data and hasn't been heard from since. 7. VISTA update VISTA news: VISTA data from Paranal has been regularly arriving each week via Garching. Each external 250 Gbyte USB disk contains, oddly enough, a week's worth of Rice-compressed data. After the usual initial delivery hiatus this mode of operation is now running smoothly and a dedicated ingest machine has been set up at this end to slurp the data off and plonk it on the VISTA raw data RAID6 array. It slurps at up to ~30 Mbytes/s and only take a few hours to plonk. There have been a few problems with missing files (TBF'd) and occassionally with missing FITS header keywords but in general data transfers are working well (10 weeks worth of disks and no problems reading them and we are on average, as expected, about 2 weeks behind the observations). ESO took over operating the telescope and ingesting data into their Paranal archive around the beginning of November. Consequently MR, with input from JRL and EGS, has set up a VISTA processing status web page in the same vein as the WFCAM web pages, which tracks data from the 1st November. This page will shortly go live on the Plone site and will, a bit further down line, have the usual DQC information available to allow external users to see some of what we are up to and monitor assorted parameters and processing status. (Earlier pre-ESO "commissioning" data is stored on-line but remains invisible to external users.) Assorted hardware foibles have delayed the SV phase for VISTA and one of them, the filter wheel positioning problem, has made generating useful science output, or even good publicity pictures, rather difficult. This was compounded by the previously reported cataloguing bug affecting all the VISTA science frames catalogues too. JRL is busy regenerating them prior to any further analysis of the CEs, throughput and sky brightness levels etc.... Although, as for WFCAM, the bug had only a marginal effect on any of these measures its a case of "if a job's worth doing it's worth doing properly". VISTA tasks: Pete was leading the VISTA effort at CASU and although a large fraction of the Christmas period was spent going through and sorting out VISTA documents, software, hardware and other things, there are still a lot of open issues related to assorted VISTA tasks that Pete was responsible for. Despite recent progress on the grant issue and Sue Cowell's (SC) appointment here for VISTA and GAIA system support, we are still seriously short-staffed for VISTA work and have had to spread some of the load around the group to get things moving. Computer support, progress pages and data ingest tasks are already in hand. Other jobs that need some TLC are: the ETC software, associated data tables and data files, and queries relating to SADT/P2PP issues which STH will take over; RWA is dealing with the data ingest and shipments of disks and will provide help with the documentation updates; JRL is the sole CPL and main ESO software support and development bod; while for now MJI will be the point of contact for science-related and general processing-related queries; JRL, STH and RWA will review the current status of the VISTA documentation set and ensure that all the latest versions are <<<< indeed up to date and avaialble on the internal web pages (MJI has hard copies of them purloined from PSB's collection). In addition to this we will all have to muck in with science and QC assement as and when it is necessary. VISTA data: STH commented that he had been asked about early access to processed VISTA test frames by various survey PIs and was tasked with telling them to get clearance through JPE before proceeding further. <<<< This reminded MJI that nothing had been heard from the narrow-banders since our last meeting and that he recalled also that Maryjn Franx had asked for, and received, some processed VISTA data taken in the proposed UltraVISTA survey region to help select the best pointing. 8. AOB MJI gave a brief update on the rolling grant saga. After much discussion with STFC a revised JES form was agreed upon and submitted mid-January and as a consequence of that a job advert for two positions at CASU was announced a week later with closing date 16th February. The meeting closed with a brief discussion of the new computer support arrangements. Continuing actions ------------------ STH check that Plone technical description pages are up to date MJI JRL produce web page of HawkI publicity for CASU MJI finish off stage-I software release pages EGS STH make photometric calibration paper available on Plone and VDFS sites MJI finish off technical paper MJI contact Mark Casali about HawkI radial distortion RGM liaise with JRL about processing HAWKI PI data New actions ----------- JRL produce a definitive solution to the pipeline versioning issues and implement it for any new processing EGS backup online 2008 ING archive data JRL redo the HAWKI NB image data processing to improve sky subtraction JRL contact JAC about possible old cfitsio version in conversion software causing ds9 reading problems for raw data JRL sort out VISTA docuementation set and make sure latest copies are STH available on internal web pages RWA STH contact VISTA PSPIs re: need to get permission through JPE for access to reduced VISTA test frames