CASU meeting Wednesday 26th October 11:30-13:00 CASU meeting area. Present: MJI, JRL, EGS, SC, MR, STH, MNO Apologies: RGM, JPE, AKY Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Other processing 10. AOB 1. Actions from last meeting ----------------------------- JRL - still continues to investigate "rolling" master linearity corrections and will include "average" channels too <<<< SC - is still fighting Ubuntu/Debian upgrade for Blades <<<< EGS - reorganised the VST data stores but haven't yet converted processed JRL data to Rice-compressed form - ongoing <<<< MJI - contacted ANS about (re)configuring the latest dual channel fibre system and indeed finally got the b*$&er functioning as required to relieve pressure on WFCAM storage STH - discussed with PCH how to split the VISTA calibration papers and MJI will essentially follow the WFCAM route - ongoing <<<< JRL - found a new home for the AAO archive contents from 20110101 -> <<<< MJI MJI - the reprocessed UDS H-band data still awaits integration into the usual processed data store <<<< JRL - checked that #55 for WFCAM catalogues does indeed record the no. of bad pixels and updated the WFCAM catalogue generator for #58 to record av conf as of data processed on or after 20111001 MJI - generated and sent off the requisite 6 monthly UKIRT Board report JRL - still needs to check if the info in darks.dqc could have detected the problem VISTA darks on 20110913 <<<< EGS - update zero-point trend plot for cf. of aluminised with silvered now that August is out of the door <<<< EGS - investigated the ESO phase III data products release and amused the assembled company by describing the "how" part, unfortunately this does not cast light on the "why" aspect given it has clearly been released with the intention of making it as unusable as possible EGS - checked that WFAU have got the latest version of VIDEO data and is in the process of relocating it to the correct resting place JRL - did indeed update our own copy of CASUrtools to fix his exptime bug JPE - inquired about upcoming VISTA PSP meeting and found that there isn't one planned, though there may be a videocon late November JRL - contacted ESO again about a more sensible VST transfer protocol, got nowhere as usual, but was superceded by events (see later) EGS - is still setting up web pages for VST processing status and access, in the interim he released the processed VST test data to the ATLAS and VPHAS PIs via a simple passwd-protected web page - ongoing <<<< ALL - progressed as much as possible all the many burning issues required to optimise VST processing, but still more to come - ongoing but split into separate new actions MJI - contacting Astrowise about their special 1 sq deg calibration regions was superceded by arranging to attend KIDS meeting in Leiden on 3rd of November where he will hopefully find out more MJI - looked into producing colour pics of Lagoon Nebula but found the data was not taken in a manner conducive to so doing - d'er ! 2. Comments on WFAU minutes ---------------------------- ....... nada ........ comments that is 3. Hardware update ------------------- One of the WFCAM processors developed problems with its RAID array which was eventually traced to the 3Ware card. After a bit of scrambling to find a supplier some spare 3Ware 9550SX cards were ordered. apm11 will have a few days well-earned rest while the cavalry arrives. MJI finally bit the bullet and after much disk migration over the summer swapped out his ancient IoA cluster Solaris desktop for an IoA cluster Linux box and is rediscovering all those best-forgotten setup quirks and foibles. A power glitch on 29th September tripped out the 3-phase circuit L1-13. Assorted machines on it needed a good poke to get them back up and running but no apparent lasting dommage. 4. Meetings and telecons ------------------------- The last VISTA IOT was 20110608 and the next will be in the future, sometime, though apparently, maybe, or even possibly, on 23rd November. MJI listened in on a VMC skypecon on 29th September but just like last time could not recall any major issues apart from some (misplaced) gripes about astrometry (see later). He will do the same tomorrow (27th October) and probably also not recall anything by the next CASU meeting. A mottely crew of not-quite KIDS folk were invited to a VST KIDS meeting in Leiden on 3rd November. Hopefully we might get some answers to the many questions that have arisen from the first tranche of VST processing. JRL is looking forward to retaining his silver star at the upcoming ADASS meeting in Paris and will attempt to progress his entente cordiale with a selection of ESO attendees. There will be a VPHAS meeting here at the IoA November 14-15 to discuss among other things commissioning and test results obtained from VST We need apm45 adding to the web monitoring setup and also today's correct new new web address. <<<< 5. Data archives update ------------------------ Everything is up to date, no transfer problems apart from the usual VST hassles. Tape backups are as up to date as they can be. The dependencies of the INT WFC DQC archive on cass02 (Solaris) disks have been terminally sorted now (cass02 RIP) and it still seems to work ! 6. WFCAM update ---------------- Ingest, checking and processing all going fine apart from minor hiccup caused by apm11 uncooperative RAID array. The July and August nights were finished off and flagged for copying to WFAU. September data is next on the list and should be completed over the next week. From 1st October, as noted earlier, processed data will now contain the average confidence level within aper3 rather than the redundant parameter in #58 that was there. Whether or not it is worth back propagating this useful information into existing catalogues to help weed out spurious outliers is an open question. The flats update for the October processing was delayed by a combination of bad weather and other circumstances so we updated those we could and started the monthly processing before the backlog became too large. Any "new" flats not used will be readied for use in November processing, which will ease the pressure on acquiring the next set of flats - the weather always knows ! MJI noted the UHS year 1 program for 2012-2013 which should add to the pressures to keep UKIRT and WFCAM running until hopefully the UHS is completed. Ever optimistic, CASU are planning for another 3 years of WFCAM processing and user support from April 2012. 7. VISTA update ---------------- Receipt of data, ingest, checking and processing has been going smoothly. Raw data up to 5th October has arrived and the last week of this is being pipelined through stage I (pawprint level). The month of August is just about ready to release to the accredited outside world and should be out of the door by the end of the week. EGS has been dutifully copying VVV data to USB disk and shipping them to Garching for onward delivery to Chile. May through to end of July is on the latest one sent. We are hoping to get some of our earlier shipped USB disks back at some point ! An update on next stage ESO Phase III deliverables for the PS PIs was received a week or so ago (they obviously read our last minutes) and repeated for reference below: December 2011 - publication of the data product format standard and Phase 3 instructions for the catalogue submission on the ESO Phase 3 web pages. Mid January 2012 - target date for submission of catalogs by the PIs of VISTA public surveys. March 2012 - target date for the publication of catalogs to the community after scientific validation by the Archive Science Group via the ESO science archive facility. April 2012 -target date for submission of images, weight maps and source lists - second survey release. Looks like catalogues only for mid-January and the rest to follow by April then. There was one query from a VISTA PI (Jean-Paul Kneib) with an observing program running on Stripe 82 region from October, who was surprised that Terapix will not reduce his data for him. http://archive.eso.org/wdb/wdb/eso/sched_rep_arc/query?progid=088.A-0728(A) En primeur Bordeaux 1st growths based on the 1855 classification will do to lubricate the pipeline engine. MJI reminded the assembled crowds that P88 started for VISTA on 1st October and the VISTA web pages need another slot to accomodate this unexpected event. <<<< Two related and particularly tiresome CASUhelps regarding supposedly poor astrometry from VMC WCS solutions were received. This took the best part of two afternoons to resolve since it required much postage stamping and stacking to attempt to replicate the problem. Suffice to say there wasn't a problem using our own software to generate colourful stacked images and the most likely culprits are bugs in ds9 and/or iraf WCS image alignment and registration. JRL recalled a similar problem raised by Robert Greimel who tracked his troubles down to a ds9 foible. On the PSF front we are planning a future trial with the long absent PSF columns. These will be based on the current catalogue positions but doing a full PSF flux fit to get more reliable photometry in very crowded fields. So it won't solve the unpicking of say globular cluster cores but it should solve some issues with aperture photometry in crowded regions. 8. VST update -------------- Its P89 OPC report time for the VST PS PIs. Our contribution, written at the same time as these minutes were being finalised, is duplicated here to minimise ... err.... duplication "CASU have been copying the raw data from ESO based on the Archive Data Retrieval requests from the PI. As this is not a satisfactory method for long term access of survey data, CASU have been investigating alternatives based on direct scripted requests from the ESO raw data Archive for all available VST Public Survey data. For routine pipeline processing this provides a more controllable data flow and also better access to more dark sky science frames for computing, for example, fringe frames. A prototype VST processing pipeline has been setup and used to process 59 nights of raw VST data (up to 20111005) taken for either general test purposes or for ATLAS (and VPHAS) project(s). Processing involves the usual steps of generating master calibration files and using them to perform: bias correction, overscan correction and trimming, cross-talk correction, flatfielding to a common internal gain system, defringing for the i- and z-bands, astrometric and photometric calibration, and object morphological classification. Jitter sequence within OBs are automatically stacked and products include catalogues of detected objects, and image and confidence maps for all frames. CASU are setting up a web-based processing progress interface modelled on the VISTA data flow system which will allow real time access to status and derived QC information. A similar PI archive access to processed data is also being constructed again modelled on the VISTA system. There is still some fine tuning required for the VST processing chain but all the main procedures are in place and working routinely." In addition to this upbeat report some interesting foibles have been spotted during the test process including: random pickup in some frames and in some detectors but quite intermittent and at several different severity levels - not easy to quantify or spot though in general fringing in the r and Halpha Lagoon Nebula frames but not on others - it took a while for the penny to drop aka bright narrow emission line covering large spatial area => fringing - unfortunately in regions like this the fringing pattern will have both detector spatial dependence and illumination source spatial dependence so quite complex and almost unpredictable fringing patterns will occur as you move the field around with respect to the optical axis interesting focus variations spotted by Hywel Farnhill and Janet Drew using the CASU catalogue QC information. MJI generated some PSF maps for a sample of these frames and rediscovered optical aberrations from having several well out-of-focus images - this was part of the shakedown process we were informed, caused by the unexpected use of filter changes during an OB - hmmmmm ..... Although much pipelining progress has occured the list of updates required is still more than enough and includes: using an rcore for the catalogues of 4.67 pixels (= 1.0 arcsec radius) <<<< the bad pixel maps need improving since they currently do not pick up all <<<< although zeroth order photometry on a nightly basis has been done and ce's derived, this needs re-running for all nights with the new ce's <<<< QC summary generating and plotting software has been updated but needs <<<< running on all nights all of the VST processed data needs squishing (a factor 5 shrinkage awaits) the classifier needs a trap for stupidly low saturation values <<<< fitsio_cat_list was updated to recognise VST data but this highlighted the need for cattype HDU parameter from CASUrtools <<<< lack of filter names in headers for special calib filter u,g,r,i makes it difficult to use - currently ignored in calibrations process web pages for VST raw data acquisition pipeline progress and summary status urgently need setting up, as does the PI access to processed data some tests on the benefits or otherwise of running the nebuliser for VPHAS cataloguing are needed <<<< finally the SV data taken on NGC253 needs a poke since there will at some point be a meeting to discuss the results from the 3 pipelines being used to process VST data (Astrowise, CASU and VSTube) <<<< Finally, finally, thanks to STH for spotting this opportunity for calibrating all VST science frames ab-initio "Data release 1 of APASS occurred on 2010 September 10, and contained photometry for approximately four million stars distributed among several hundred discrete fields between -10.15 < Dec < +87.35. Data release 2 occurred on 2011 February 10, and included an additional four million stars from the southern hemisphere, acquired from our site at CTIO. Data Release 3 occurred on 2011 August 10, and further extended southern hemisphere coverage; the catalog now contains photometry for 18.9 million objects in about 27,000 square degrees. The figure below shows the coverage, with RA 5 hours at the front. Dots mark the center of each field." JRL has already grabbed a copy of the latest catalogue and MJI will investigate its use for calibrating VST data <<<< 9. Other processing - HAWKI, INT WFC, MegaCam, Subaru, VST, CTIO - UVEX, IPHAS, and Kepler processing All quiet on the Eastern Front for a change 10. AOB The ESO in-kind process has become somewhat unclogged of late and there is even a chance it might start in the New Year. Continuing actions: ------------------- JRL - investigate benefits of a "rolling" master linearity correction or even an average channel one SC - progress Ubuntu/Debian upgrade for Blades EGS - squish the burgering processed VST data to release ~4/5ths of storage JRL STH - progress the VISTA calibration papers writeups JRL - use the new home for the AAO archive from 20110101 -> MJI - integrate the reprocessed UDS H-band data into processed data store JRL - check if the info in darks.dqc could have detected the problem darks EGS - update zero-point trend plot for cf. of aluminised with silvered EGS - setup some web pages for VST processing status and access New actions ----------- SC - add apm45 to the web monitoring setup and figure out today's address JRL - get P88 added to progress web pages for VISTA JRL - use an rcore for the catalogues of 4.67 pixels (= 1.0 arcsec radius) JRL - update and preferably improve the bad pixel maps EGS - rerun photometry with new ce's EGS - and run the QC summary software now it exsits JRL - trap stupid saturation values generated in classifier JRL - add a cattype keyword to HDUs for imcore in CASUrtools MJI - do some tests on the benefits of running the nebuliser for VPHAS MJI - poke, or better still stack, the NGC253 SV data MJI - check the efficacy or otherwise of APASS for VST calibration