CASU meeting Wednesday December 1st 11:30-13:00 CASU meeting area. Present: JRL, MJI, STH, EGS, AKY, JPE Apologies: MR, JCR, NAW, RGM Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. Optical/NIR processing 7. WFCAM update 8. VISTA update 9. AOB Minutes ======= 1. Actions from the last meeting -------------------------------- STH - have remeasured the colour equations and updated the plone web pages MJI these revised ce's are now being used and new ZPs will be back-propagated to all VISTA data for the next revision release (which will also include a few other minor header information updates). The extinction- dependence is still being finalised as are the updates to the Calibration Plan - ongoing <<<< EGS - put WFCAM illumination correction tables on the web - ongoing <<<< MJI RGM - the catalogue listing software is now being widely used and any bugs or upgrade requests are now being dealt with via the usual channels EGS - updated the catalogues with the zero-points from the new ce's and MJI delivered them to the ESO SV teams MJI - WFCAM calibrations in regions of high extinction investigation is STH still ongoing <<<< MJI - still needs to update the summary tables total exposure time column to account for the occasional missing dither component. <<<< JRL - duly dribbled and that piece of fine code is now tooled up. Related to this was a brief discussion about the centralised repository for the binaries for use at CASU. AKY - stacked the UltraVISTA season#1 OB outputs and found nothing untoward, EGS however JPE noted that at the PhaseIII meeting H_Joy_M claimed to have found artefacts around bright stars with his deep stacks. AKY will liaise with H_Joy_M and cf. stacks. <<<< JRL - sent details of the object masks required for use in sky creation during pipeline processing, and has so far heard nada mas ALL - after a bit of Badgering, attendance for the Christmas lunch was sorted 2. Comments on WFAU Minutes --------------------------- A brief discussion of the speed of executing the nebuliser on various machines ensued. JRL noted that on the Blades stacked pawprints took about 3-4 mins to nebulise per CPU but as there are 10+ quad-cores available for VISTA processing this is not a bottleneck when pipelining. Appreciation of tiling issues much .... err ..... appreciated. The Grouting software (see section 8.) developed after a bit of mulling apres les dernieres minutes, appears to sort the problem of 96 variable PSFs and after a few more tests will be trotted out to update ALL the tile catalogues, since it appears to be harmless in cases when the tile catalogues are ok already. It was noted that VIDEO stacks have been delivered to WFAU. Some discussion on who is doing what where why when and to whom followed. VMC, VVV and VHS, and even UltraVISTA, have been very good at keeping us in the loop, the best that can be said about VIDEO and VIKING is that they haven't. 3. Hardware Update ------------------ The Supermicro RAID6 arrays apm44_a and b are now rocking and provide 64 Tbytes of usable storage. The system has been designed for at least one JBOD expansion module of similar storage ilk, i.e. the infrastructure is already in place. An expansion module will be purchased in the New Year as soon as we have tidied up the rack use to make adjacent space. The apm23_c RAID (holding WFCAM raw data) blew one of its two power supplies a while back. These PSUs are no longer available, and since the blown one appears to be unfixable, this ancient 7TB RAID system will be retired in the near future. Meanwhile its contents have been slurped over to another RAID system as a backup. One of the ports of the dual fibre-channel card in apm23 went walkabout thereby removing apm23_c and apm23_d from visibility. Fortunately these astoundingly ancient (3 yr old) cards are still (just) available and one was duly purchased and fixed the problem in two shakes of a pointy thing. The Blade systems still suffer occasional hangups including connection issues with their local scratch processing space. The long term plan is to switch these over to Ubuntu to help? this and other NFS issues. 4. Meetings and Telecons ------------------------ Quite a few this month: JRL and MJI attended a fruitful and interesting ADASS meeting in Boston (7th-11th November). Highlights for MJI included the Herschel presentation and a Belgian Beer Festival, whereas JRL just loved the discussion about the future fate of the ADASS Conference Proceedings. MJI went to Durham for a VST ATLAS meeting on 23rd November to take part in a discussion of the the end-to-end processing system for the ATLAS VST data e.g. pipelining, calibration, QC control, archiving. CASU will be responsible for the 2D processing, catalogue generation, astrometric and first-pass photometric calibration, and pipeline QC issues. The overall photometic calibration, based on tile overlaps, will be handled by Steve Maddox and WFAU will produce the OSA (sic!) and make sure Tom does his QC checks. Most of the science data taken will be for the 3 Public Surveys, ATLAS, VPHAS+, and KIDS. Rather than just process bits of the nightly data, CASU would prefer to process the whole lot, since its virtually no more effort and would yield better calibration opportunities, particularly generation of nightly fringe frames. MJI will initiate a cunning plan to achieve this by first contacting Konrad K. to see if a suitable deal can be struck. <<<< A lengthy VISTA IOT telecon was attended by MJI, JRL and JPE. There were two or three nuggets worth reporting (although the Turkish Delight provided by AKY was much more appreciated): One of the main areas for discussion was the reliability of GSC2, UCAC3, 2MASS, USNO ...... as SADT input catalogues for VISTA guide and wavefront sensor stars. Issues here are: point source (or otherwise) reliabilty, accuracy of magnitude estimates (I-band is the relevant one), proper motion issues, completeness and depth issues ..... usw. An internal ESO report presented in the papers prior to the telecon summarised the relevant features for all current choices. Although GCS2 is not bad, the best, i.e. most accurate astrometrically, most complete, and most reliable by a mile is 2MASS. To make it so though requires a decent I-band predictor from the 2MASS J,H,Ks. MJI volunteered himself and STH to investigate the transformation between 2MASS J,H,Ks and the VISTA I-band, a sensible colour selection criteria to avoid the reddest most extinguished stars, and the level and form of any extinction-dependent correction term. <<<< Problems with acquiring guide or wavefront sensor stars are not disconnected to the other significant discussion concerning the VISTA observing effiency. In this regard it was agreed during the telecon to reduce the frequency of standard fields being observed per night to a minimum (i.e. <=3 per night). In parallel a reassessment of the observing overheads is taking place. Additionally, there are plans to do some "few-off" observations of bright spectrophotometric standard stars in Z and Y (particularly) to provide a an absolute flux calibration for these bands. Fortunately sanity prevailed, and rather than introducing a dodgy windowed r/o mode it was decided to try the defocussing-the-telescope route first. MJI sat in on two VMC telecons (28th October and 25th November) since the previous CASU meeting. Among other things he noted that the PSF photometry, from the deep stacks of the 30-Doradus region being done by one of the consortium members, will provide a good comparson with our usual processing software and the VMC stacking being done by WFAU. These telecons also give a chance to point out various processing features that are not yet common knowledge e.g. discussing why the aperture photometry works so well when compared to PSF photometry even in relatively crowded VMC fields. There was also a question asking what the MJD of the stacked pawprints and tiles referred to - nice one ! The short answer is, it is the MJD of the first image used in the dither stack and/or stacked pawprint making up the tile. The longer answer is the provenance information in the headers contain all the tracking required to figure out either the mean MJD of say a stacked pawprint, or if you really want to do this properly the weighted mean MJD of each catalogue object (or even pixel for the truly pedantic) in a tile (this is required due to the stacked pawprints all being taken at different MJDs). JPE, RGM and EGS attended the ESO phase III meeting held at Garching - see section 8. below for more details but meanwhile to quote JPE, "Monika will be the VST Marina". Upcoming meetings include: the VVV meeting in Vina del Mar, Chile, December 8-10th, Phil Lucas and Eamonn Kerin are going. After his return Phil and Maren Hempel will be calling by 16-17th December to discuss processing issues and grab a few USB disks of VVV data. the UKIDSS RAS meeting in London, January 6-7th, MJI is thinking of going to this but would like to see an agenda before finalising the VMC science meeting in Munich, January 18-21st, MJI is planning to go and asked if anyone else was interested in this or the others above <<<< 5. Data archives update ------------------------ AAO, ING, WFCAM, UKIRT, VISTA et al. all trucking along fine so not much to report here apart from an imcopy/cfitsio 16- -v- 32-bit compression incompatability issue that gave EGS a mild twitch, followed by a satisfied smile when he figured out what was causing it. 6. Optical/NIR processing -------------------------- Everything motoring along nicely, EGS commented on a few issues with the CTIO SERVS data he is processing, which thanks to the readout mode, which uses only one amplifier (rarely used), required him to recompute the cross-talk matrix (the last one we had dated from 2002). At least fixing cross-talk in optical mosaics cameras works ! 7. WFCAM update ---------------- MR reported via email: We are up to date with the data transfers (JACH & ESO), and we are up to date with the processing (all November nights are either finished or under processing). All October data has been flagged and WFAU has almost completed the transfer (3 nights left). MR received a few requests for WFCAM raw data that were gently redirected to the ESO archive. Only one project is in the reprocessing queue, the good ol' UDS H-band data spanning 05b-07b, about 60 nights in total. The UKIRT board met about a week ago, no news is ..... no ..... news. 8. VISTA update ---------------- Ingestion and processing is falling into a steady routine, with releases of processed data for the preceding month occurring early the next month (capisce?). We currently have received raw data up until November 17th, this is almost fully processed barring two nights which are still running. MJI felt we should investigate the question of the evolution of the flatfields (and possibly darks) with time: specifically concerning systematic noise/feature changes, and rms noise properties. AKY leapt at the chance to justify our monthly flatfield update policy. <<<< MJI raised the "Gas Mark 4" issue pertaining to data from the 5th/6th October some of which was found to be unprocessible. This was triggered by the fact he could not find any examples in the processed directories. There was a good reason for this and JRL will plonk the list of affected (i.e. rejected and not processed) frames in READMEs in the relevant processed directories. <<<< The processed NB975/985 observations that Matt Jarvis took in DDT time were made available a few weeks ago but no word about their quality etc... has dribbled by. AKY reported on the ongoing saga of OB grading versus inherited image keyword headers: the documents ESO send us to report their OB grading does not necessarily contain correct information and does not necessarily report the status of the whole OB but sometimes its pawprints. For example, there are cases where the OB is aborted due to some technical problems and then continued to be executed and completed. The ESO grade page gives a status "Aborted" for the first 3 stacks and "Completed" for the last 3 stacks, and we set the grade of the whole tile as "Aborted" because the tile takes its headers from the first image. Also, there are OBs which don't have any grade information or the reported times in the ESO grade page are not correct. Hence there is no way for us to find out correctly the status of some of the OBs. In the end, it is ESO's responsibility to make sure they send us the correct information so that we can put it into the headers. Meanwhile because the pipeline is smart enough to combine the correct parts of multiply attempted OBs and form a "completed" product the question arises of whether we should flag this in the header, or just leave it to the QC information derived from the products to allow the end-user to decide if the OB makes the grade required for science delivery. There followed an interesting discussion on tile grouting - viz whereby you make the tiles look nice after plonking them on the wall. [The fitsio_grout programme takes as input the tile catalogue. Providing all the relevant stacked pawprint catalogues and confidence maps are somewhere to be found, it then computes differential aperture corrections at the location of each detected object, weighted by the confidence map that was used to drive the tiling of the image. The individual detector aperture corrections (#1-7), 96 of them for each aperture, define the differential aperture corrections i.e. the difference with respect to the median for the whole tile. All fluxes and associated errors for apertures 1-7 are corrected, larger apertures are negligibly affect by seeing variations and are currently left alone (ditto the other fluxes). The updated catalogue then has to be re-classified and re-photometrically calibrated. The correction software optionally can fix aberrant individual input detector aperture corrections and also adjust for changes in MAGZPT.] EGS has been testing the grouting on VHS data and reported dramatic improvements in the CMDs and two-colour diagrams for the ~20% of the data affected by large seeing variations during the OB. This manifests as either multiple stellar locii or very broadened stellar sequences. After grouting the incidence of problem tile catalogues drops by two orders of magnitude while leaving the other (good) 80% of tiles unFU'd. EGS will look at the impact of grouting on a sample of VVV data to extend these tests to more traumatic fields before we commit. <<<< Assuming all goes well, this procedure can be tacked on the back-end of the tiling pipeline. The question then arises of do we keep pre- and post-grout tile catalogues or just toss the former. The overwhelming consensus in the room was that given that the tile catalogue is already not directly traceable from the tile image, there is no point in keeping the un-grouted tile catalogue. As warned a while back, the tile products are not yet v1.0. With the grouting procedure in place, and assuming remaining tests go well, the plan is that all existing tile catalogues will be grouted, re-classified and re-photometered. This will then become the v1.0 tile release. Finally, EGS and JPE reported back on the ESO public surveys phaseIII meeting for VISTA PSPIs held at Garching yesterday (Tuesday). Documents and various presentations are available from http://www.eso.org/sci/observing/phase3/workshop2010.html In particular the Phase 3 User Documentation (http://www.eso.org/sci/observing/phase3/workshop2010/p3userdoc1.1.pdf) describes the ESO policies, timeline, data product formats and data submission process (much amusement was had by all in the ensuing discussion about this, in particular the data product formats, file naming convention and some of the required keyword changes). Short summary: It is the responsibility of the PI to prepare, validate and submit the data products for each survey, but he or she can delegate the submission process to a co-I or list of co-Is. The PI is however the ultimate responsible for closing a release. ESO defines t0 to be April 2010 (i.e. start of science operations) from where releases are expected. At t0+12 months (i.e. the first occasion is April 2011) the PI has to release tiles and confidence maps, together with "source lists" e.g. CASU catalogues, and source variability catalogues and light curves if required. These releases happen every 6 months. At t0+18 (i.e. the first occasion is October 2011), the PI has to release deep stacked tiles and catalogues (with magnitude columns !) and band-merged catalogues (ditto). This is done yearly. The format of the catalogues, and how to make them VO compliant, is not yet fully defined. The release process is described in the document above and a tutorial was carried out at ESO. Note that ESO requires certain data formats (i.e. particular filename extensions, particular keywords) which in some cases conflict with the CASU format. The data upload to ESO is carried out using lftp (or any other ftp client which supports sftp). Several validation procedures are carried at ESO before ingesting in the archive during which the PI can be asked to update the release according to the ESO guidelines. Catalogues will be ultimately available from ESO in a RDBMS to allow for simple queries. Small comments to the document, if any, are welcome. 9. CASUHELPs ------------- ..... were rolled over until next time 9 and 1/2. AOB ---------------- It's slippery out said 3-Gaia'ites who couldn't walk from A to C without going via their B. Continuing Actions: =================== STH - produce "final" VISTA colour equations extinction dependence and MJI finish revising calibration plan document EGS - put WFCAM illumination correction tables on the web MJI MJI - finish WFCAM calibrations in regions of high extinction STH MJI - update summary table software for total exposure time mod AKY - liaise with H_Joy_M to cf. results for season#1 Ultra VISTA data stacks EGS New Actions: ============ MJI - will contact KK about processing VST KIDS data STH - investigate the transformation between 2MASS J,H,Ks and the VISTA I-band ALL - express interest or otherwise in upcoming meetings