CASU meeting Wednesday 28th September 11:15-12:45 CASU meeting area. Present: MJI, JRL, EGS, SC, MR, STH, JPE, MNO Apologies: NAW, RGM, AKY Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Other processing 10. AOB 1. Actions from last meeting ----------------------------- AKY - is now on maternity leave so we decided to drop the cf. of deep stacks as an action AKY - prior to going on leave she revealed the location of the flatfield report aka http://casu.ast.cam.ac.uk/documents/reports/vista/ vista-flatfield-report-v2/files/flatf_report_v2.pdf JRL - still continues to investigate "rolling" master linearity corrections and will include "average" channels too. This is worth nailing and should eliminate spurious gain variations that keep being noted in the Garching pipeline, but not ours due to sheer user brilliance. <<<< SC - progress further Ubuntu/Debian upgrade for Blades which is paused but not forgotten <<<< EGS - finished checking how often the WCS convergence problem occurs for VISTA by comparing # of matched stds for each detector etc .... and found that it didn't. MJI - scrummed down and finished off (not) sorting out some of the middling EGS ancient problems with the ING archive since the data transferred from JRL ING in the period of dodgy DVDs, viz 1999/2000, are unrecoverable JRL - put to sleep the previously fixed broken catalogues saga and had AKY contacted the relevant PIs EGS MJI - sorted out the APM catalogue server binaries and moved them to the PC cluster from whence they can be found via the usual web pages EGS - inquired about the EVALSO transfer and was informed that there would be no more EVALSO transfers (or equivalent) for another 3-4 months while the system is fully integrated into a Black Hole MJI - discovered that the WFCAM standdown was from June 28 to July 22 MJI - during the standdown the GPS photometric recalibration was completed (see WFCAM section for more details) MJI - likewise the long-standing WFCAM H-band UDS reprocessing was done 2. Comments on WFAU minutes ---------------------------- ....... nada ........ comments that is 3. Hardware update ------------------- Much tidying of disk storage, particularly for WFCAM raw data, has occurred. The new server for VST, apm45, was spec'd, cost'd, ordr'd, delivr'd and installed and is rapidly filling up with assorted VST test data for ATLAS, VPHAS and SV programmes. All we need now is to organise it better and process it properly. <<<< The latest dual channel fiber system has been tested ok but it still refuses to configure as a single RAID6 device. MJI will contact ANS and see if they can cut through the manual obfuscation. <<<< APM29 dropped a disk - clumsy - and rebuilt - replacement disk est arrivee. Rodents: the usual seasonal infestation occurs de temps en temps. Saga so far mice 3-0 casu; bait 2-0 mice; rematch fibres chewed -> deceased mice nailed under the floor, extinct, no more - maybe. 4. Meetings and telecons ------------------------- The last VISTA IOT was 20110608 and the next will be in the future, sometime. MJI listened in on a VMC skypecon on 28th July but could not recall any major issues. He will do the same tomorrow (29th September) and probably also not recall anything. The NIR-Optical Galactic Plane meeting in Herts (July 18th-22nd) was attended at various times by MJI, NAW and STH. who delivered talks on VISTA processing time series analysis and VST commissioning data results. The two most common questions/misunderstandings related to photometry of saturated stars and calibration. Conclusions: - for saturated stars don't use tiles due to the mishmash of different detector saturation levels, while pawprints are better, shorter exposures would really rock'n roll - we need to write the VISTA calibration papers as soon as possible given the semi-infinite number of ways the community misunderstand the issues and the equally large number of questions sent to CASUhelp about it <<<< JRL reminded us about ADASS in Paris this year commencent 6'ieme Novembre MJI pointed out the upcoming ROE Autumn meeting Royal 10-12th October on "Astronomical Simulations for Instruments & Telescopes" and asked if anyone was going - apparently not. 5. Data archives update ------------------------ JRL commented that the AAO archive was running out of space. He and MJI will figure out what to do about it. <<<< Apart from that everything else is up to date, no transfer problems (apart from VST - see later), and the raw archives continue to be backed up to tape as the data pours in. The dependencies of the INT WFC DQC archive on cass03 (Solaris) disks have been sorted by moving all interfaces with this archive to the PC cluster in readiness for retiring cass03 and replacing it with a Linux box. 6. WFCAM update ---------------- During the WFCAM standdown from June 28 to July 22 MJI took the opportunity to fulfill two long standing requests for reprocessing. The UDS H-band data for 56 nights scattered throughout 05b-07b was finally reprocessed and sent off for testing. Results look good and this will be integrated back into the processed data repository after a bit of header massaging. <<<< The other painful task was to revisit the GPS calibration in an attempt to make the photometry more robust in regions of high extinction. This was a two-fold operation involving deriving updated aperture corrections and revisiting the photometric calibration with an improved way of dealing with high extinction zones. After much deliberation it was decided to update ALL GPS photometry and ALL other WFCAM photometry to make it all consistent. The only (sic!) changes were to catalogue headers, not contents, involving parameters such as seeing elliptic saturate and aperture corrections, followed by recalibration which changes magzpt magzrr numzpt nightzpt nightzrr nightnum and added ebmvmed - the "average" computed E(B-V)'. The relevant image headers were also updated. After all this excitement normal processing resumed early August after a new set of flats was available. The standdown was to investigate among other things the intermittently sticking J filter paddle and resulted in the more infrequently used Z slot being switched with the J as a further precaution. Since then transfers (apart from the odd glitch) and processing have gone smoothly with no sign of the sticking filter problem. The iceman briefly returned on 20110901 with a completely iced up cryostat window after a passing sonic screwdriver reversed the polarity on the dry air device and made it a moisturiser, but only for the 1 night. There have been large numbers of requests for instant access to processed data the last few months. Although not a major issue to deal with, the numbers of these have been steadily rising over the last year or so, particularly from the "paying customers". During DR9 checking SJW reported a few oddities in the processed data which was tracked down to inconsistent interleave jitter offsets that resulted in a stack of two LAS superframes requiring different offsets for the 4 detectors [not allowed in the robustified processing adopted a few years back]. This feature has happened a couple of times before, though has proven hard to pin down exactly what causes it. SJW also asked about nos. of bad pixels affecting objects and the possiblity of using that sort of information to help weed out spurious outliers. It was pointed out to him that catalogue parameter #55 already contains this information. This then reminded MJI of a question regarding the (useless) WFCAM catalogue parameter #58, which for VISTA is used to record the average confidence level for an object (useful for spotting problem regions). JRL was asked to update the WFCAM catalogue generator to populate #58 with the same info, starting 1st October. <<<< This also raises the question of back-propagating the #58 info into the existing WFCAM catalogues, doable by simply interrogating the relevant bits of the confidence maps. Tom Kerr (re)raised the issue about WFCAM processing given the loss of one of the cameras. The current pipeline sidelines any nights where this has happened. After a brief discussion the simplest solution would be to operate a modified processing pipeline for those nights rather than complicate the existing working pipeline. A long overdue tidying up and re-arrangement of WFCAM raw data disk storage was also completed, hopefully not breaking too many web links. Last but not least, yet another Board report is due by 26th October for the December meeting. <<<< 7. VISTA update ---------------- There have been the usual odd few foibles such as: hexaopod leg control (pas) for M2 due to faulty power supply near the end of June stopping VISTA observations below 55 deg elevation, mainly of interest to VMC; and the odd ESO archive-end grumble about missing VISTA images due to OBJECT, OCS and DPR keywords being absent. The latter hasn't affect our processing and all the files are here so we have ignored it and left them to it. The only other event of note involved one of the VISTA controllers causing a temperature problem on 20110913 for #13-16 during acquisition of calibration data (darks). This showed up as -ve skies in some frames after processing but was quickly spotted during checks and darks from an earlier night used to reprocess the night correctly. MJI asked if the QC checks on the darks (cf. with WFCAM procedure) could have picked this up before the night was processed. JRL will investigate. <<<< Ingest, processing and releases are trucking along nicely. Raw data keeps appearing on disk ~2 weeks behind the observations being taken with the latest up to 14th September appearing during the week. Processed data is running as usual ~another 2 weeks behind this, with the month of August being dusted off for its final iteration. Data up to end of July was released mid-September. MJI asked if we had an updated zero-point trend plot to get a better idea of the difference between the aluminised and (pristine) silvered mirrors. ESG said he would produce an aluminised one to end of August when that month is finished with. <<<< The VISTA Phase III deliverables are now available through the ESO Archive Query Interfaces for Phase III ingested data products http://archive.eso.org/cms/eso-data http://archive.eso.org/wdb/wdb/adp/phase3_vircam/form http://www.eso.org/sci/observing/phase3/data_releases.html It was noted that ESO "urge Ultravista & VIKING to complete their Phase III submission at the earliest". MJI asked EGS if he could browse how ESO have made the products available, since he (EGS) delivered the bulk of them, and see what fun we are missing. <<<< There has been complete silence from ESO regarding the next batch of Phase III deliveries which were due at the beginning of October. Unsurprisingly all the PI's are keeping their heads down on this one. MJI asked about the status of shipping disks of VVV data to Chile. Basically, this was awaiting enough data May -> July to fill one up. EGS has this in hand and has arranged to send one via ESO Garching in the next week or so. We think the reprocessed VIDEO data has been exported to all interested parties and are about to relocate it to its final (correct) resting place. <<<< JRL and NRC converged on noting that stacking of very different exptime exposures revealed a bug in the CASUrtools stacker which caused garbage output in these situations. For normal use there was no problem, so hopefully this bug has minimal impact. JRL was tasked with updating our own copy of the binaries. <<<< Apparently there is a VISTA PSP meeting sometime in the Autumn to review survey progress (and deliverables?). JPE was tasked with finding out if and when and how. <<<< 8. VST update -------------- Acquiring VST data is proving to be an unbelievable pain as it is based on interactive web logins to pick up whatever bits of data ESO see fit to allocate to us. Without getting a whole night of data it is impossible to organise a sensible filenaming system and a pipeline processing scheme. JPE said that he believed VST data were piped in almost real time to Garching, and thence to Astrowise. JRL will contact ESO again to see if we can organise a more sensible transfer protocol. <<<< All of the VST data we have managed to transfer so far need reorganising into a raw and processed night-based scheme and converting to our usual Rice-based storage protocol. JRL and EGS will sort this out before we start setting up some web pages for status and access. <<<< There are many questions to be addressed here. Is the pointing good enough for the "tile" overlaps being used; what is the astrometric distortion; what are the calibration options; colour equations; what is the early VPHAS, ATLAS and all the NGC253 SV data telling us about the science capabilities of VST ? <<<< Apparently there are 3 pipelines being used to process the SV data (Astrowise,CASU and VSTube) is there going to be a meeting to discuss comparative results ? Too many questions, but one thing's for sure the VST User Manual is a tad optimistic given the lack of detector-level filter names in headers for for the special calibration filter u,g,r,i - d'oh ! There are photo opps (e.g. Lagoon Nebula) but an interesting question is <<<< access to the special 1 sq deg calibration areas that OmegaCam have set up, MJI will attempt to find out more. <<<< To add to the pressure there will be a VPHAS meeting at the IoA November 14,15 to discuss the VST test results. 9. Other processing ------------------- Aka HAWKI, INT WFC, MegaCam, Subaru, CTIO, UVEX, IPHAS, and Kepler At least we have dropped VST from the above list. The main action has been in completing the Kepler (and UVEX) processing for 11b, which just leaves the UVEX and IPHAS backlog from 2010 observations. 10. AOB ------- CASUhelps keep on rolling in but no time to amuse bouche the tasty bits. One interesting item that makes support for CASUrtools seem like a good deal. Various people, VVV in particular, want to do catalogue completeness tests - excellent! let them make it so. A bit of news: Jenny left at the end of July to pursue a new career in Medical Imaging. While Aybuke is off on maternity leave Martin Niederste-Ostholt (MNO) has been recruited to help out with the VISTA processing. Meanwhile the lawyers are getting stuck into delaying the ESO in-kind work until the next Millenium, which is fine with us given the lack of personnel here. Finally, its time for the Beer tasting ........... Continuing actions: ------------------- JRL - investigate benefits of a "rolling" master linearity correction or even an average channel one SC - progress Ubuntu/Debian upgrade for Blades New actions: ------------ EGS - organise the mess that is the VST data store and convert to JRL Rice-compressed form MJI - contact ANS about (re)configuring the latest dual channel fibre system STH - discuss with PCH how to split the VISTA calibration papers with the MJI baseline expectation being the WFCAM model JRL - find a new home for the AAO archive MJI MJI - integrate the reprocessed UDS H-band data into processed data store JRL - check that #55 for WFCAM catalogue does record the no. of bad pixels and update the WFCAM catalogue generator for #58 MJI - generate the next 6 monthly Board report JRL - check if the info in darks.dqc could have detected the problem darks EGS - update zero-point trend plot for cf. of aluminised with silvered EGS - investigate the ESO phase III data products release EGS - check if WFAU have got the latest version of VIDEO data and if so relocate it to the correct resting place JRL - update our own copy of CASUrtools JPE - inquire about upcoming VISTA PSP meeting JRL - contact ESO again about a more sensible VST transfer protocol EGS - setup some web pages for VST processing status and access ALL - progress the many burning issues to optimise VST processing MJI - contact Astrowise about their special 1 sq deg calibration regions MJI - produce colour pics of Lagoon Nebula