CASU meeting Wednesday February 23rd 11:30-13:00 CASU meeting area. Present: MJI, AKY, EGS, SC, JCR Apologies: STH, NAW, RGM, MR, JPE, JRL Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. Optical/NIR processing 7. WFCAM update 8. VISTA update 9. AOB Minutes ======= 1. Actions from the last meeting -------------------------------- STH - the VISTA colour equations and extinction dependence has been sorted MJI for now, required before we can sign off the calibration for v1.1 of the data products. The residual scatter in regions of high extinction had been proving troublesome and is improved somewhat though still no sings until fat cigar lady the STH - have checked yet again possible causes of wfcam -v- vista MJI delta magnitude slopes and found that they vanish to negligible levels if due allowance is directly made for vista <-> wfcam ce transforms STH - the WFCAM extinction calibration investigation is likewise parked MJI MJI - updated the table software to take account of the possibility of NICOMB being different to NDIT for individual pawprint extensions AKY - liaised with Henry from UltraVISTA and found that though the artifacts around bright stars have gone away, acquiring an example of their deep stack image to investigate other issues would be good news - ongoing <<<< MJI - has contacted KK about processing VST KIDS data but heard nada STH - produced a colour transformation predicting i-band mags (for guide stars in SADT) from 2MASS JHKs photometry. It looks like we can predict i-band mags to within 0.1mag given 2MASS JHKs. STH has robustified the simple linear fits and ignored the extinction dependence. Now it just needs putting on the web pages <<<< AKY - is still investigating the evolution of the flat-fields with time; looking at systematic noise/feature changes on different scales, and rms noise properties - ongoing <<<< JRL - have found out what's afoot at vvv.survey.org; lots of colourful EGS pictures mainly but no unexpected gotchas from the pipelined data EGS - will complete v1.0 -> v1.1 process for data up to end of March 2010 by AKY February 28th and flag it as ready to go. The remainder, i.e. until MJI end of November 2010, will take another week after that and will be flagged likewise. Once this is done, December and January v1.1 data will also be released - ongoing <<<< As part of this process detector-level zero-point offsets and illumination corrections have been computed for October 2009 through to November 2010. The latter should be made available via the <<<< CASU web pages for interested parties and checked for any signs of obvious trends <<<< 2. Comments on WFAU Minutes --------------------------- No comments 3. Hardware Update ------------------ The Supermicro 45-bay JBOD expansion module is still sat on the floor due to an incompatibility between SAS1 and SAS2 - an updated 3ware card will be installed to deal with this. Rather worringly the Supermicro host system spotted the fact that we peaked inside the lid (to check out the existing 3ware card for the external JBOD) and told us off. Another fibre-channel RAID system was ordered and delivered to replace (eventually) the limp-along ancient RAID system with one unique power supply remaining. 4. Meetings and Telecons ------------------------ The next VISTA IOT will (tentatively) be on 14th April. This has been pushed back due to the telescope down-time (4th March - 7th April) while the mirror(s) are (re)aluminised. There's a VMC Skype-Con tomorrow (Thursday 24th February) for which MJI will don the headphones and listen in. The VVV/VPHAS/EGAPS meeting at Uni.Herts. will be held the week from 18th July 2010 -> topics: Galactic Plane; Galactic Plane; & Galactic Plane MJI reminded folk of this year's NAM which will be held in Llandudno from 18th - 21st April. Based on years of family days out to the aforementioned place, JCR recommends a visit to 'Happy Valley' on the Great Orme which has pretty views, a cable car, toboggan run and dry ski slope. The name says it all really! [Great Orme ? or Happy Valley ? - you decide] http://www.greatorme.org.uk/Trail9.html 5. Data archives update ------------------------ All going very nicely thank you. 6. Optical/NIR processing -------------------------- Nothing much to report, so either proceeding nicely too or not at all 7. WFCAM update ---------------- The weather at UKIRT has continued in its grumpy vein but the good news is that while cleaning the ice off the cryostat window some local extinction was removed too improving both the zero-points and the K-band sky brightness. Receipt of data, and processing and delivery of the usual ESO subset of raw data, is up to date. Release of processed data to WFAU is a bit behind due to a pile up of other things, but in migitation processed data has been promptly released to PIs as the usual multiple queries about early access to the processed data are received from vigilant PIs. A new joint JAC-CASU-WFAU Memorandum-of-Understanding was circulated and agreed on recently. An updated set of Pipeline and Archive Science requirements is also doing the rounds (still). The status of these and many other fascinating facts can be found on the JAC/management website. 8. VISTA update ---------------- We have received new disk-delivered data up until 9th February 2011. Data from 1st December 2010 through to January 31st 2011 has been processed using the v1.0 pipeline and is ready to put through most of the v1.1 upgrades and now only awaits the monthly detector zero-point updates from MJI <<<< After the release of v1.1 data from October 2009 to November 2010, these latest two months will also be finished and flagged as ready-to-go. EGS has been monitoring the internet transfer of data from ESO to CASU following JPE's request for a test on the efficiency. So far he has transferred one month of data and noticed a variable transfer rate - between 0.9 Mb/s to 2.5 Mb/s per thread. With 8 threads running in parallel this delivers 12-16 MB/s (or 0.7-1 GB/min if you prefer). There is no obvious correlation of rate with other factors like time-of-day and so on, but the overall transfer speed is very promising. MJI asked EGS and AKY to check if anything is missing in the internet-transferred data cf. to the usual disk deliveries where we have overlap. AKY has <<<< already made use of the internet data to compute new flats for February processing since the current disk set did not have enough to do so (and the next delivery has gone on vacation in Paris). There followed a brief discussion summarising the re-processing steps to morph v1.0 -> v1.1 to make sure we hadn't forgotten any. To recapitulate, this involves: * re-running the classifier on all paw-print catalogues; * re-running the photometry on all paw-print catalogues; * generating monthly detector zero-point updates (and illum corrs); * FITS header updates on all paw-print catalogues/images; * running the grouting code on the tile catalogues; * followed by tile catalogue re-classification, re-photometry; * and finally header updates on tile images or to summarise, no change to images, just header updates, some changes to catalogue table contents and headers. This solves (we hope) the majority of problems with v1.0 data and if that proves to be the case we will migrate to a WFCAM-like only fix the-broken-files philosophy rather than globally reprocess everything. Before he went in hospital for his well-earned rest, JRL implemented the MJD update to tile catalogues as part of the grouting process for v1.1. An extra column of information, MJD-int(MJD) is now recorded in column #63 for each object. The int(MJD) information is recorded as MJD_DAY in the HDU. The column actually records the difference in MJD minutes so that the optional differential MJD FITS map is easier to browse (and also very artistic with the correct LUT). Examples and info about this will be added to the web pages. <<<< All v1.0 tile catalogues have been saved for now - pawprint catalogues are all header changes so there is no point in saving them as they can be reverse engineered if necessary. Regarding Phase III delivery, we will deliver the CASU flat file products for VIDEO, VVV and VHS to ESO. The deal with all PIs for us to do this is that they have to supply us with a list of tile product filenames (CASU style) to send. Whilst on this topic MJI commented on the difficulty occasionally of correctly interpreting the quality of an OB based solely on tile QC information. As an example if say 4 of the 6 pawprints making up a tile OB had good seeing and/or nicely circular images and/or good throughput (zpt), while the other 2 were completely out of spec with poor seeing or elliptical images or rubbish throughput, the odds are that the reported tile QC parameters would probably look ok since they are based on robust estimators. The only reliable way to guarantee overall tile QC is to make use of the provenance information in the headers to track the QC information in the pawprint stacks making up the tile. He said he would add a note to this effect on the relevant web pages. <<<< There is still some ongoing debate with ESO about what form and what content band-merged products should take. However, as it is unlikely CASU will be generating band-merged products to send to ESO this should not impact on us too much. As noted earlier VISTA will be down between 4th March and 7th April while the mirrors are (re)-aluminised. After all the v1.0 -> v1.1 kerfuffle a well-earned oiling of the software cogs will ensue before getting to grips with our contribution to Phase III data product delivery. 9. CASUHELPs ------------- If MJI had realised how much agro saturated stars would cause he would never have written the how-to-bodge-the-magnitudes article d'oh! Another recurring gem is how do I combine n single FITS files to make a multi-extension FITS (MEF) file and its pal, how do I extract single FITS files from MEFs. Answer: you guys and gals need to talk to each other, or learn a simple scripting language having something like plugin CFITSIO modules. There was an "interesting" query questioning the efficacy of the CASU morphological classification based on their comparison of deep stacked VIDEO data (using Sextractor to produce the catalogues and classification) to CASU VIKING products. Answer: (polite version) the classification returned by Sextractor is not perhaps as good as it might be. AOB --- Nada - at least nada than we can report in polite society ........... >>>> Continuing Actions: =================== AKY - if possible acquire example of deep stack for direct cf. with ours AKY - continue flat-field properties investigations and report at next meeting EGS - finish off the v1.0 -> v1.1 process; have ready the first 6 monthly AKY tranche by the end of February and the remainder ~1 week later. New Actions: ============ STH - put 2MASS JHKs -> i-band predictor information on web pages EGS - make the illum corr info (plots only) available on the web pages MJI - examine detector zpt offset trends for October 2009 -> current period. MJI - generate the December and January detector zpt offsets plus illum corrs EGS - compare data from disk and internet transfers. Identical? or not? AKY MJI - add MJD tile catalogue info to web pages and example pic MJI - update QC/grade note on web pages about pawprint QC -v- tile QC JRL - make a speedy and full recovery and dust off those {} brackets