CASU meeting Wednesday 23rd November 11:30-13:00 CASU meeting area. Present: MJI, JRL, EGS, SC, MR, STH, MNO Apologies: NAW, RGM, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Other processing 10. AOB 1. Actions from last meeting ----------------------------- JRL - still continues to investigate "rolling" master linearity corrections and will include "average" channels too <<<< SC - is still fighting Ubuntu/Debian upgrade for Blades <<<< EGS - the Rice-compression for the processed VST data is done JRL STH - progress with VST calibration papers slow and still ongoing <<<< MJI - the reprocessed UDS H-band data still awaits integration into the usual processed data store (WFAU have only just copied it) <<<< JRL - still needs to check if the info in darks.dqc could have detected the problem VISTA darks on 20110913 <<<< EGS - updated zero-point trend plot for cf. of aluminised with silvered and has included up to end of September EGS - set up the web pages for VST processing status and access SC - has not yet added apm45 to the web monitoring setup but did figure <<<< out the current address JRL - P88 has been added to progress web pages for VISTA JRL - is now using VST catalogue rcore of 4.67 pixels (= 1.0 arcsec radius) JRL - updated and even improved the VST bad pixel maps EGS - reran the VST photometry with latest ce's EGS - and ran the VST QC summary software JRL - trapped stupid OmegaCam saturation values generated in classifier JRL - added a cattype keyword to HDUs for imcore in CASUrtools MJI - ran some tests on the benefits of running the nebuliser for VPHAS not much difference was the conclusion but if VPHAS wants it then it will be made so MJI - stacked a selection of the NGC253 SV data and made some diagnostic CMDs and a colourful image MJI - checked the efficacy or otherwise of APASS for VST calibration and concluded it would be useful as a secondary check on things like illumination correction but probably not quite deep enough as a primary calibrator 2. Comments on WFAU minutes ---------------------------- ........ nearly went monthly but segued back to fortnightly ......... 3. Hardware update ------------------- Getting spares of the 3Ware 9550 card for apm11 has proven more challenging than usual but a couple of spare cards were finally procurred and one of them duly inserted to cure the recalcitrant RAID problem. VST web pages, raw and processed data and most of the processing are hosted on apm45 => it is time to order its big brother, apm46 <<<< Weird scp problems on apm28 and apm36 were tracked down respectively to an innocent shell change from bash to tcsh in the login script and sourcing an lheasoft startup script - d'er! All hell may break loose with the upcoming IPV6 changes to the University network which are slated to be implemented in the next month or so. After much discussion and impenetrable possibilities, a laissez faire approach was suggested, aka - wait until it breaks before fixing it. After recovering access, at least temporarily, to the network monitoring pages MJI then grumbled about the temperature flatlining - either the temperature control is fantastic or some recent software upgrade broke the monitoring - take your pick. SC will investigate. <<<< 4. Meetings and telecons ------------------------- MJI attended the VST KIDS meeting held in Leiden on 3rd November. The only other non-KIDS participant invited (sic!) was Tom Shanks. Both of us gave talks on ATLAS related things plus a summary of the assorted foibles that CASU have uncovered during VST processing so far (a more detailed summary of MJI's notes from the meeting are in the VST section). JRL went along to ADASS, in Paris, 6-10th November and came back enthused and replete with gossip, and also with his silver star intact. There was a VPHAS meeting in Cambridge 14-15th November which JRL, EGS attended, and MJI part attended. There was much discussion of observing strategy and the results from the test data taken so far (any choice bits are incorporated in the VST section). There was an action therein to compile a foibles list and make sure ESO are aware of the problems. <<<< The next VISTA IOT telecon is scheduled for 1 hour after this CASU meeting finshes. MJI, STH and JRL will attend - watch this space. The next VMC skypecon is tomorrow, MJI will listen in, in case it morphs to a grypecon. There is an upcoming VST ATLAS meeting in Durham on 5th December. MJI will attend for CASU, but he will be outnumbered by WFAU representatives. An example of VST processed data was made available to WFAU to give them something to mull over in advance. The future of UK Submillimetre Astronomy Workshop is at the IfA 12-13th December - much relevance for the future of JCMT, and UKIRT, but unfortunately nobody from CASU can make it. MJI will attempt to get a learned summary when he visits IfA later on the 13th for another meeting. <<<< 5. Data archives update ------------------------ Everything is up to date, no transfer problems, and even the VST transfers have been automated thanks to EGS. This reminded MJI that we need to start tape backups of the raw VST data acquired so far. <<<< 6. WFCAM update ---------------- Ingest, checking and processing all going fine apart from the ongoing minor hiccups caused by the apm11 RAID array for processed data and a problem with Kauwa at the JAC end which interupted the transfer of raw data for 201111030 for a short while. The processed September nights were finished off and flagged for copying to WFAU on 2nd November and October looked to be heading the same way until a minor faux pas with one night was discovered during final checking (again caused by the apm11 RAID array). This night, 20111009, has now been sorted and the October data will be ejected from the building shortly. As noted earlier the reprocessed UDS H-band still awaits dumping to its final resting place but other than that there are no significant items currently in the reprocessing queue. Eric Gaidos contacted WFAU about his "missing" transit project data for 2 nights in August. After searching in vain for it they in turn contacted us. The short version of the story is that the data for this project is deliberately taken out-of-focus and flagged with RECIPE='QUICK-LOOK' i.e. do nothing, which the pipeline duly did. Rather than simply dumping the raw data somewhere for him to pick up it was processed (de-darked, flatfielded, and decurtained) off-line and then made available. It wasn't possible to sky correct since the sequences of ~150 frames for each night were taken in ~the same sky position and this was the only 1-0S data taken on those nights. 7. VISTA update ---------------- Receipt of data, ingest, checking and processing has yet again been going smoothly. Raw data up to 9th November has arrived and data up to the end of October have been processed through Phase I of the pipeline. September data was released to PIs and WFAU earlier this week and the first grumbles, in this case about 2 missing tiles arrived within 24 hours. Both missing tiles, on separate nights, had suffered from trailed images at the start of the pawprint sequences. Athough all 6 stacked pawprints required for the tiles were created (and look very good - a testimony to the pipeliners) the tiles weren't. The reason is probably related to the rejected frames at the start of the OBs. JRL is investigating further. <<<< Meanwhile MNO will generate the two "missing" tile offline. <<<< The P88 web pages were generated as noted earlier, but with a minor faux pas in the page header. EGS will fix. <<<< Whilst checking the September photometry during the detector/illumination phase, MJI spotted a 0.1 mag jump (improvement) in the ZPs between 20110909 and 20110910. This turned out to be due to cleaning of the lens top-end which was suffering from both dust accumulation and condensation. The trend plot on the technical pages, which now includes the September data, shows the jump very clearly and also reveals that the throughput in all bands (apart from z, but that is to be expected) is back to September 2009 levels ! 8. VST update -------------- The first pass analysis of the VST data from 16th August to 5th October, which also formed the v0.5 test release to PIs, showed up several obvious pipeline issues that needed fixing in addition to highlighting several genuine problems with VST data. Version 0.9 which should address the main image processing issues is well underway, and should include processed data up to the end of October by the end of this month. However, calibration issues, virtually all related to the photometric calibration of the catalogues still need to be addressed. We expect that focus, rotator and other tracking problems will gradually decrease from their current 5-10% level over the usual bedding in period for new facilities. The current QC information generated during processing is adept at spotting these problems anyway. The focal plane of OMEGACAM is curved and the VST shows field-dependent astigmatism (none in the center of the FOV) such that - if you have got the focus right - you get very small ellipticities elsewhere, but a measurable gradient of the FWHM. If things are out of focus, then you may get relatively large ellipticity (astigmatism) in the corners (depending on the sign of the defocus) or you may get the best FWHM in the corners and no astigmatism - should keep the QC checkers on their toes. The other obvious image-level problem is down to filter positioning repeatiblitiy, or rather lack of. Fortunately for the monolithic ugriz filters this is a minor problem only affecting the outer long sides of the 4 corner CCDs. However, the segmented 4 quadrant filters NB659, including also surprisingly B and V, have a large masking cross in the middle which not only makes them more susceptible to filter positioning issues, but also to light scattering off the mask. Unfortunately the apparently random occasional gain variations seen in a few specific detectors (1-5%) and the more serious scattered light problem (10+% over the field), noticed during various photometric and image tests, require much more effort to characterise and attempt to find solutions for. The gain variations show up in master flatfield ratios, as does some of the scattered light problem. Dark skies, made from conventionally flatfielded science images, show different background light patterns suggesting two main causes: non-astronomical light entering the optical train from various sources other than direct sky - the dome is a very open construction with the biggest impact being on twilight flats that will invariably suffer from different asymmetric scattered light pollution everytime they are taken; scattered light from sky due to multiple reflections within the optical system - these should be mostly radially symmetric but the effect appears to be quite large ~10%. We are currently investigating different ways of characterising the scattered light more quantitatively, since by its nature it generates an equivalent illumination correction requirement. Without correction the calibration does not even satisfy a 10% limit, even in perfect photometric conditions. Plan A. was to use the 8 roughly equally spaced equatorial SA 1 sq deg calibration regions that the OmegaCam consortium are setting up to generate an illumination correction map. These are being made from sequences of (seriously) dithered VST observations. Unfortunately this requires having to solve for any illumination corrections, due to scattered light and other causes, at the same time as everything else. This is hard to do and nowhere near available yet. So we need to probably dust off Plan B. then. The good news on the calibration front is that if we can solve the illumination correction problem, even with only the 3 SA's observed each night that form the primary calibration plan, we are finding virtually all nights yield a very steady ZP measurement. ZP variations per pointing are well within 5% and certainly good enough for a first-pass global calibration. The other good news is that the raw VST calibration and public survey data have all been made available directly from the ESO archive since the start of VST surveys on 15th October. This also includes data taken during the science verification and commissioning period from 16th August to 14th October. Using a bit of ingenuity and smidge of scripting, EGS has managed to concoct a way of automatically transferring the raw data over the Internet. The only gotcha is that although all VST data headers are available in the archive ~1 day behind the observations, data is shipped on USB disk and take 1-2 weeks to appear. Unfortunately the archive gives no indication of whether or not the data is there (sic!) so we are running the script weekly ~2 weeks behind the observations. [Running it daily lead to complaints !]. From the available complete ESO observing logs, EGS and JRL have also figured out how to leave gaps in our filenaming convention for non-survey data taken during any nights e.g. GTO programmes. This means that if future requests come in for processing non-survey observations we can stick with the in-place data/run no. filenaming for the already processed data. 9. Other processing - HAWKI, INT WFC, MegaCam, Subaru, VST, CTIO -------------------- - UVEX, IPHAS, and Kepler processing No news - either good or means we have been too busy with other things. 10. AOB ------- There was only one item, the CASU christmas party. STH volunteered to book somewhere for lunch around mid-December and organise emailing the troops, and debt-collecting. <<<< Continuing actions ------------------ JRL - investigate benefits of a "rolling" master linearity correction or even an average channel one SC - progress Ubuntu/Debian upgrade for Blades STH - progress the VST calibration papers writeups MJI - integrate the reprocessed UDS H-band data into processed data store JRL - check if the info in darks.dqc could have detected the problem darks SC - add apm45 to the web network access monitoring setup New actions ----------- MJI - spec, get quotes, and order apm46 SC SC - see why remote temperature monitoring is flatlining MJI - compile VST foibles list and liaise with ATLAS and VPHAS PIs re: action MJI - acquire learned summary of ROE workshop on submm et al. future JRL - generate tape backups of raw VST data EGS JRL - figure out why tiles were not generated in two cases MNO - generate the burgering missing tiles EGS - fix faux pas in VISTA P88 name header STH - organise the CASU christmas party