Wednesday 15th July 2009 APM Meeting tables 11:30-12:30 Present: MJI, STH, RWA, NAW, MR, EGS, JRL, SC, RGM (later), DWE (later) Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST..... 6. WFCAM update 7. VISTA update 8. AOB - rolling grant status etc.. Minutes ------- 1. Actions from the last meeting RWA VISTA technical pages are now being populated with technical EGS information derived from various commissioning data and from drafts of reports from JRL. Strange formatting is being eliminated in parallel and a link to the pipeline progress web section is being added. Since this will be an ongoing activity for at least the next several months and the information is also needed as input for the VISTA User Manual that ESO are preparing, this item has now been taken off the action list and will in future appear in the VISTA update section. MJI stage I software release - still ongoing <<<< MJI technical paper - still ongoing <<<< MJI utility to convert binary tables plus corrections - still ongoing <<<< EGS UVEX support - program sent MJI A/C unit fixed after nearly 2 weeks - a stunning response time. MJI & SC have re-configured apm35_a to be the raw VISTA data suppository and are in the middle of converting apm29_a,b,c to be the VISTA processed data dump. JRL will look at PSB's stuff on apm29_b before we get rid of the partition and see if we need to keep any of it. The processed data on apm29_c will be moved to apm29_a prior to getting rid of the apm29_c partition and tidying up the rest of the detritus. Related to this STH suggested we construct a guide to what data is on what disk (mainly related to raw ING archive it turned out) and felt so strongly about it that he was volunteered to update the relevant web page and also to keep it up to date. <<<< MJI the Petrosian Fix (sounds like a techno-thriller) still needs to be carried through but has been held up by assorted VISTA commissioning tasks and the finite number of hours in a week. <<<< JRL has submitted an abstract for ADASS XIX and thereby gets to keep his Silver Star and his horse MJI made the data for the assessment of the need for filter flushes on WFCAM available, but does not know if any subsequent decisions have been made. He suspects it will be left up to the individual survey heads and PIs of other WFCAM programmes. STH have compared the ETC throughput estimates with those derived by STH MJI by hand and found good agreement. VISTA appears to have nearly a factor of 3 higher DQE compared to WFCAM. These throughput values will shortly be included on the technical web pages. The ETC estimates of sky brightness also seem very good. Todos still are the magnitude limits/signal:to:noise estimates and an update of the overheads. There is now enough data to get a good estimate of the average sky brightness in Z,Y,J,H,Ks for the ~month of data in hand since VISTA restarted on-sky observation on 19th May. MJI will generate an overview summary plot of the sky brightness levels over this one month period and hand it over for the technical web pages. <<<< 2. Comment on WFAU minutes No significant comments so we had a resume of status of data transfers, DR6 checking and timescales. 3. Recent and upcoming meetings The last VISTA IOT on June 30 produced some interesting numbers and dates. The minutes are on the CASU internal web pages for those of a curious bent. The next IOT telecon will be July 29+/-1. The deadline for abstract submission for the UKIRT workshop (14-16th September in Edinburgh) is the end of July. STH said he intended to go to give his invited talk but just in case he is observing then he has a backup plan to delegate to David Pinfield. MJI, in collaboration with Annette Ferguson, will be submitting an abstract on the LG work with WFCAM. There will be a VDMT this afternoon. 4. Raw data archives AAT,ING,WFCAM,VISTA everything ok and up to date. Now that a lot of the dust has settled MJI asked JRL and RWA to look into creating an offline tape backup of the raw VISTA data using the LTO4 library unit attached to apm29. <<<< UKIRT - JRL said he was back on the case and promises to finish by late summer, though which year wasn't specified. 5. Optical/NIR processing : HAWKI, INT WFC, MegaCam, Subaru, VST...... Nothing much happening for a change mainly due to hols and effort dedicated to processing VISTA commissioning data. EGS is processing some Subaru data and STH is tackling some CFHT and `bizarre' Russian data - whatever that means. 6. WFCAM update. Processing: MR noted that apart from the odd hiccup in raw data transfers due to knock-on effects from a power failure at JAC end on 20090623, everything is running smoothly. Raw data transfers are up to date as is the processing. June data has just been flagged as ready for transfer. Reprocessing: All the GPS nebulosity reprocessed (more accurately optional extra) data for DR5 was flagged as ready to copy and has been slurped up. Likewise the UDS K-band data earmarked for the UDS DR5 release has also been flagged and transferred up by Edinburgh. DR6: The Petrosian Fix is still next in the queue, it got bumped by VISTA data temporarily, but should be out of the door by the end of July. Various UKIDSS checkers have reported back: LAS jpeg checking looks good - only 1.1% FU's and these are virtually all trailed images; GPS noted a small bunch of moon reflection issues on one particular night 20081113. The "shadows" change rapidly with frame sequence number making them effectively unfixable, but given the nature of the shadowing it probably doesn't affect the catalogues much anyway. MR cunningly spotted some ZP funnies with some UDS data on 2080920. It transpired that the pipeline had obviously got bored with interleaving first and then stacking and instead went straight for stacking. All the data passed QC checks since there was nothing wrong with it per se. This problem occurred about half way through the night. A few GCS frames were also affected in the same way. It looks like this occurred after the pipeline was restarted mid-way through the processing. Impossible to reproduce the problem, never been seen before and hopefully won't be again. We'll write that one down to the "fat electron" theory. The affected frames have already been reprocessed and reflagged to go. 7. VISTA update Raw data has been received via Garching for up to and including June 24. It includes some daytime calibration data with good linearity sequences from which JRL has also derived some good bad pixel masks. JRL also exploited the recent linearity data to investigate low light level linearity. Some of this data was taken to explicitly probe the low-count regime and indeed shows non-linearities exist at the 1 - 3% level even below 1000 ADUs. STH commented he needs to revisit the linearity check in the catalogue photometry -v- 2MASS to ensure that the non-linearity has indeed been corrected for. He also needs to revisit the colour equations now that we have about twice as much data as last time we computed them. <<<< As noted earlier, MJI will supplement this with the sky brightness estimates and plots. EGS independently discovered that the standard for describing WCS keywords in FITS binary table catalogues is different to what we have been using so far. Some of our post-processing software has not been adapted to the different standard yet. JRL, as always, is taking a broom to the pipeline and tidying things up and fixing things faster than he breaks them. A pernicious buggette with the pipeline astrometry which generally manifested itself in crowded fields may have been tracked down and fixed, much to the post-pipeline astrometry fixer's relief. This may also have been the cause of some of the ESO tickets about the stability of the photometric calibration results from the summit and Garching pipelines. In contrast the Cambridge photometric zeropoints on the summary pages look very well behaved. JRL released via the VPO a new version of the summit pipeline to fix a couple of problems but does not know if it is being used yet. He reported that Sandra Castro, who was dealing with the VISTA pipeline has left ESO and is being replaced by Lander de Bilbao. During processing recently apm29_a,b,c hung up a couple of times. It may be a software issue and SC has now applied the most recent software patches to see if that fixes the problem. MJI noted that since the upgrade the remote temperature monitoring web page for apm29 has flatlined. SC will take a look see why. <<<< Remaining commissioning tasks include investigating the efficacy of deep stacking on a 16hour set of J-band pawprint data and completing the loop <<<< for the offsetting accuracy when/if we get more data. The other previously outstanding issue relating to stability of the pin cushion distortion model, including centre coordinates, was booted out the door last week. Having been asked to look at science data, MJI examined the VISTA Y and H tile OBs for the globular cluster M55 and produced an appealing colour image from the tiles together with a plausible CMD from the individual detector stacks. He noted that the passbands used had been cunningly chosen to avoid all published theoretical isochrones so it was difficult to analyse the rather nice M55 locus quantitatively. Feeling upbeat he then tried to repeat the trick on test tile OB J,Ks data taken of the Sculptor dwarf (to compare with CIRSI observations) at airmass of ~1.5 and found that subtly varying seeing throughout the tile exposures caused some interesting problems when using catalogues generated from the resulting tiles. This was not completely unexpected since each sub-region in the tile is made up of different partial exposures potentially leading to nasty disjoint PSF variations across the tile. The effect was worse than anticipated even using aperture photometry, which is not as susceptible to systematics as PSF fitting. This highlights a generic problem in doing accurate photometry on VISTA tiles and will in general require a VISTA-specific solution. He then outlined a couple of possible solutions neither of which is particularly elegant. This requires some further work. 8. AOB MJI reported on the current grant status and noted the recent joint letter that had been sent out by WFAU & CASU to alert users to the situation. SC & MJI are looking at ways of adding new A/C capability to deal with the proposed increased amount of Gaia hardware and also in the near future at least to provide some redundancy in cooling capability. This posed the ancillary problem of what to do with the large number of (mainly) ING archive CD/DVD storage cabinets which would need to be moved out to make room for new computing kit. Continuing actions ------------------ EGS add catalogue software and binary table conversion software to software MJI release pages MJI finish off technical paper updates MJI write utility to convert binary tables with all corrections MJI do the Petrosian fix STH further compare ETC fact with fiction MJI New actions ----------- STH update data storage web page particularly with respect to ING archive MJI generate an overview summary plot of the sky brightness levels using recent VISTA data and hand it over for the technical web pages RWA generate LTO4 backups of VISTA raw data received to date STH revisit average zero-points and colour equations for VISTA and check non-linearity correction is working by cf. with 2MASS data SC fix the flatlined temperature monitoring web page for apm29 EGS investigate stacking on the 16hour set of J-band pawprint data