Wednesday 1st June 2010 APM Meeting tables 11:30 - 13:00 Present: MJI, EGS, JRL, RGM, SC, JPE Apologies: NAW, MR, JCR, STH Agenda ======= 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent & upcoming meets 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST 6. WFCAM update 7. VISTA update 8. AOB Minutes ======== 1. Actions from the last meeting -------------------------------- EGS stage I software release partially done but now awaiting JRL to JRL to investigate building a release distrubtion more robustly <<<< STH updating the Calibration Plan document is still ongoing <<<< JPE raised the issue of filter keywords for the not_the NB980/990 filters (975/985) at the VISTA IOT. As expected just a single keyword in the PHU is present and don't hold your breath for propagation down the SHUs. Gavin "remembered" which set are which - honest ! and for the record 1-8 are NB975 and 9-16 are NB985. JPE cunningly split the difference by labelling the whole lot NB980 in the only filter keyword value. All we need to do is find a redshift 7 object to confirm which is which, on second thoughts better make that OII or Halpha at lower redshift. RGM is still testing the catalogue conversion software to destruction <<<< EGS text for our own versions of ESO-released PR images on the CASU web ALL pages needed. EGS still needs to put in links/cross references to the ESO versions all ongoing <<<< EGS finished off the SV reprocessing and delivered the goods in time STH for the Orion meeting. MJI JPE raised the VISTA CVRAL issues at the IOT telecon and JRL sent in a minimum opus re: his thoughts on the subject. MJI constructed and sent off the UKIRT Board report. He will attend the meeting next week. JRL fixed it -> apm29_a disk space and minor IAMFUD FUDs in the catalogues 2. WFAU minutes ---------------- Not many, other than RGM asked what the schedule for the next UKIDSS release was. 3. Recent and Upcoming Meetings ------------------------------- Way too many: EGS and JRL partook of a VVV group meeting via telecon. Most things seem fine. There were a few grumbles about missing files and supposedly the CASU server was unreliable at times, neither of which we can find any clear evidence for. MJI was entertained by the contortions (seven separate steps) that one of the VVV members was going through to generate (nice looking) tiles. It would be interesting to do a comparison against our single step procedure tiles; EGS will contact them and make it so. <<<< VISTA IOT telecons: they do drag on a bit. At 3+ hours the last one has to be one of the biggest wastes of time on the planet. There are a few perennial problems which no matter how much chin wagging always seem to come back The CRVAL problem + associated other features; and the persistent use of sparse sampled non-deblended cataloguing to estimate ellipticity (and seeing) QC parameters and infer misleading trends. The biennial detector zero-point issue raised at previous meetings also seems to be a facet of the QC pipelines being used rather than a genuine problem. MJI produced and sent round a bunch of histograms from the science pipelines from April data that look fine. Persistence is futile cropped up again for discussion but the conclusion (pushed hard by CASU) was that if a sensible observing sequence is used persistence is not a problem and does not need specifically correcting for since any remnant images dither stack out. Finally, JPE undertook to politely inquire if we can have more focussed discussions at future IOTs. <<<< NAW, STH and MJI (intermittently) took part in the PLATO Data Centre meeting at the IoA 27/28th May. Good progress was made in defining the nature of the processing tasks and organisational structure. IoA leads the Exoplanet Analysis System development. Fingers are crossed for next hurdle. Meanwhile RGM attended the EUCLID consortium meeting in Paris on 25/26th May. There was discussion about Ground Segment Data Processing which includes the processing of imaging data from the Optical imager and NIR imagers and spectra from the NIR slitless spectrograph. Mellier from IAP and Pasian from INAF led the discussions which were mainly high level. There will be further discussion with ESA about the division of responsibilities and deliverables between ESA and the Euclid Consortium. The next stage has a deadline in early September, 2010. The second CASU release of the SV data products based on v0.8 processing occured a few weeks ago. EGS and STH attended an SV Orion meeting at ESO to explain the processing, discuss v0.8 issues and even see what sort of science is being done. The SV NGC253 team maintained a respectful silence other than grumbling about persistence problems in their SV data, cunningly sequenced such that even in deep stacks all the remnant images line up perfectly. EGS and STH produced a set of notes made during the meeting that are included as an Appendix (for internal access only). There was an IoA review of computer needs a couple of weeks ago at which MJI and SC made sure the WFCAM and VISTA computer requirements were safely accounted for. MJI is off to ESO again this week to further the progress, or otherwise, of the ongoing negotiation with ESO re:in-kind payments. Colin Vincent, Bob Mann and Keith Noddle are the other UK reps. Michele Peron, Fernando Comeron, Pascal Ballaster, Andreas Kaufer and Martino Romaniello turned out for ESO. MJI pointed out the upcoming meeting on Galactic near-IR survey science being organised by Melvin Hoare, in Leeds on Thursday 8th July 11am-6pm "UKIDSS and VISTA Galactic surveys meeting". JPE and he are going. 4. Data Archive --------------- WFCAM, ING, AAO, UKIRT, VISTA all up to date and trucking ....... 5. Optical / NIR Processing --------------------------- Not a lot going on other than a pile of Subaru PF Hamamatsu detector data being processed. VST - there is an upcoming "Review of VST Public Surveys" September 28-29 2010 in Garching. Though the letter quite carefully states that the "ESO STC has asked us to revisit the scientific goals of the approved public surveys with VST" rather than review. The cast list includes the public survey and GTO PI's, the original PSP, and the corresponding "experts" of STC (Y. Mellier and R. Nichol) hmmm...... MJI and EGS will probably partake for CASU to support Janet and Tom's endeavours. 6. WFCAM Update --------------- Processing and raw data ingest is up to date. WFCAM went on vacation from 24th May, back on the 11th June we believe, for a well-earned spot of surfing while its host is having its mirror re-aluminised. The May data is being checked and flagged off even as we speak. No news from enterprising PhD student after he got all the raw WFCAM data for CMP/3. The last of the parallax measures for T720 were ticked off and made available to the eager team. CASU are glad to see JAC humour persists with the new "FOC_TOFF" keyword. MJI noted the new arrangements for UKIRT with Tom Kerr becoming Head of UKIRT Operations and that Chris Davis will remain our point of contact for WFCAM matters. The reprocessing queue has grown with Phil Lucas' usual request for nebulous things in selected regions. We will add that to the small pile of incumbents. 7. VISTA Update --------------- Ingestion and processing data to pawprint level (version 0.8) up to date and done and dusted up to 23rd April VISTA intervention. A modest amount of v0.9 has been included due to the need to reprocess parts of a few selected October and November nights where either the sky subtraction did not get done or detectors #5 and #6 were swapped (19th November) and needed unswapping prior to processing. As noted earlier the v0.8 SV reprocessing was finished on time and the science products distributed to the SV teams. This includes all the standard flat files including tiles and is intended as a thorough external shakedown of the system before releasing v1.0 of everything shortly. Version1.0 reprocessing has begun with the 1st month of data from 15th October in progress. This includes, as part of a throrough shakedown test, the automatic production of tiled data products. After yet more discussion on this we have opted for a slight sea change again: the catalogues will be produced from the nebulosity filtered tiles but the images will be the unfiltered tiles - reasoning behind this on application. RGM pointed out that there was a web page on the ESO website that specified required keywords for processed imaging data products. http://archive.eso.org/cms/eso-data/data-submission/image-guide e.g. Photometric magnitude system (AB, VEGA) PHOTSYS Whilst these are Survey PI requirements and can be added later or existing keywords values copied, he thought it might be more convenient if they were included in the CASU products. The consensus was that after these have stabilised, or indeed been added to, we should revisit this issue since it is also not yet clear how ESO will deal with catalogue data product FITS binary tables. Though having said that, interestingly, the DICB, who oversee FITS issues at ESO have not complained about either image or catalogue keywords produced by the pipeline (other than some totally trivial things - see previous minutes). JRL tried out the algorithm that uses externally generated sky masks on the Fynbo January NB118 data with improved sky subtraction results. Unfortunately this whole sky masking issue involves knowing an accurate WCS much earlier in the pipeline and has required a rewrite (gulp!) of the dither stacking front-end. On the +ve side this also helps with the CRVAL/rotation matrix problem. Let's hope its all +ve since there are some potential gotchas with sharp teeth lying deep in left-field. The short wave suicidal mania of #16 has been assessed further in making tiles for the SV programmes. Currently we have not altered the confidence maps prior to tiling since in practice we found it better so far to deal with the problem at either tile catalogue generation or interpretation phase. Tests on this are ongoing with more datasets. JPE noted that despite ESO being happy with the delivered VISTA pipelines they have still not been formally aacepted. He will contact Michele and Fernando and attempt to get this signed off. <<<< The data transfer test is still on the agenda and will likely involve the transfer of a week of compressed data from Garching to Cambridge (in addition to transfer tests from Paranal to Garching). Oddly, ESO seem pleasantly surprised by the 4x Rice compression obtained. JPE also raised the need to reprocess the PR data from 28th September to 14th October since assorted bods have asked for access to the FITS images. This is not completely straightforward since various bodges with sky subtraction were needed due to the interesting way observations proceeded. 8. AOB ------- Not much for a change other than MJI finally got his wish with some fitted external fans for the main computer room to supplement/replace some of the unnecessary cooling - energy consumption spies please note. EGS sent an email around earlier about the recent Plone website upgrade and asked for any faux pas to be reported. JPE was first on the draw, long may it be so. Continuing Actions: =================== EGS - finish adding catalogue software to release pages JRL STH - finish off last bits of calibration plan document update JRL RGM - test catalogue conversion software ALL - contribute text to go with CASU versions of VISTA PR images EGS add links to ESO versions New Actions: ============ EGS - contact VVV team and get an example tile for comparison JPE - politely inquire about more focussed discussions at future IOTs JPE - contact ESO re: signing off pipeline deliverables