CASU meeting Monday 30th April 11:30-13:00 CASU meeting area. Present: MJI, JRL, AKY, SC, STH, JRF, JPE, EGS Apologies: MR, RGM, NAW Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- JRL - still continues to investigate "rolling" master linearity corrections including "average" channels <<<< SC - completed high-five Blade Ubuntu/Debian upgrade STH - made zero progress with VISTA calibration papers but promised to circulate draft versions in time for next meeting <<<< JRL - checked if the info in darks.dqc could have detected the problem VISTA darks on 20110913. First discovery was that it was in reality 20110813! The problem was clear in the diagnostic tables so a warning has been added if the background sigma rises above a certain threshold. This has been tested on some good darks and some dodgy ones. JPE - investigated, post-meeting but pre-minutes writeup, if the location JRF of VISTA diffraction spikes with respect to bright stars are readily predictable. They are but are so numerous that flagging them individually is impractical and they are instead looking at better defining the radial zone affected -v- magnitude. <<<< RGM - the timing and content of phase3 deliverables to ESO is still in a state of flux so the action on which VHS data to deliver has been dropped for now SC - is still chasing up ordering the new G-bit switch <<<< EGS - the vulnerability with apm28 has been addressed, that of apm25 is SC more problematic and awaits one of either apm47 or apm49 to be up and running as a temporary archive server <<<< SC - investigated the nfs whinges and concluded it was due to heavy i/o load SC - ordered and installed a replacement B/W printer for apmhp4 JRL - acquired the latest version (v5) of APASS and MJI has put it to use MJI - confirmed that the raw UHS data will not be going to the ESO archive MJI - the UKIRT Board report was duly written and sent off EGS - regrouting to fix VISTA apcor2 problem on hold for now AKY - has caught up with the VST processing backlog JRF - investigated if the VISTA processing web pages could reflect the full processing status but the complexity of VISTA processing makes this a more subtle problem, so this item has been shelved for now 2. WFAU minutes --------------- MJI noted the recent comment about low transfer speeds and wondered if any progress on the cause had occurred or if the problem had gone away. 3. Hardware update ------------------- Assorted Gaia goodies were delivered and installed since our last meeting. Once all the new stuff is signed off this will free up a spare LTO4 unit and a 30TB disk array. apm48 was duly installed and the HAWKI and VIMOS imaging data transferred across as a test. The 3TB disks turned out to be nint(2.73TB) - d'er! apm49 est arrivee and awaits installation together with apm47 the apache server on apm3 was upgraded to fix yet another vulnerability issue and the system kernel upgraded too home disk space was getting a bit tight so various sundry old accounts were cleaned up and moved elsewhere. Are these login accounts still needed ? <<<< STH's transient alert machine is still burblin' away on the floor, moving it to the table next to it would be a better location while it awaits its Infiniband card prior to Dog Kennelling it. 64-bit versions of various odds and sods of calibration software were created are any more needed ? A couple of the Blades seem to be having sporadic I/O problems unrelated to the disks. SC will monitor/try assorted upgrades.... firmware.... <<<< The X11 libraries and ds9 are missing from apm45. SC will investigate. <<<< 4. Meetings and telecons ------------------------- JRF and MJI went off to NAM to sample the culinary delights of Manchester, and the odd beer. The last VMC skypecon was on 5th April but nobody present at it could recall anything of note. The next one is due 3rd May. NAW and MJI attended the WEAVE meeting near Amsterdam on 17th-18th April to discuss the CASU data management role in this project. The same odd couple went to the VPHAS/IPHAS/UVEX meeting at UCL on 23rd-24th April. MJI focussed on the foibles of VST data processing and calibration and demonstrated the intricacies of devising a suitable illumination correction for the 0.25 mag of scattered light. JRL went to dry off in Nice for the GaiaESO meeting held there 18th-19th April. Much discussion about data products and formats ensued and even some first results from the science data flow. MJI reminded everyone about the UKIDSS final workshop at the end of May/ beginning of June and the rapidly approaching deadline to register. <<<< 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ Everything is up to date though the UVEX suppository and dispenser needs a bit of tidying up. In lieu of the need to revisit the U-band calibration the tidying up can await re-creation of the band-merged products. MJI and EGS will then rationalise the serving up of UVEX data. <<<< 6. WFCAM update ---------------- Acquiring raw data, ingest and processing are all proceeding apace. The final monthly calibration for March (and now April) still needs doing and the data releasing to WFAU. <<<< A UKIRT engineering and maintenance shutdown is planned for 21st May - 6th June. Apparently the dates are driven by the new Korean group who need a new filter installed, plus their targets are up in June/July. This also coincides with the main switchover from UKIDSS to UHS. It also provides a convenient slot to back-propagate the av conf column upgrade to the earlier (i.e. most of the) WFCAM data. MJI will make it so over this period. <<<< 7. VISTA update ---------------- Receipt of data, ingest, checking and processing has been going smoothly since the switchover to Internet-only transfers. The process of catchup for VISTA processing has begun but will take a bit longer to achieve than for the VST processing. Raw data up to the end of April has arrived and data up to the end of March have been processed through Phase I of the pipeline. The January and February VISTA data were postprocessed for calibration, pushed through the Phase II pipeline and released to the PIs et al. The ESO missive on Phase 3 deliverables arrived via one of the PI's. It makes an interesting read, with two bottom lines: we can't deliver anything for now; and the next delivery required will be band-merged catalogues, variability tables and similar ilk, aka ........ "The next Phase 3 submission will consist of the catalogue products only, so that a better support can be provided to the users and to the entering into operations of the Phase 3 catalogue facility." Most of this is not down to us either ! Meanwhile VHS are agitating to release more images and cats as is VMC who want us to deliver some OB-level products, ESO willing. MJI pointed out the lack of a P89 entry on the progress web pages plus a few leftover "ingested" but not correctly flagged odds and ends, including what appears to be an unprocessed night with valid data in aka 20120215, emphasising the importance of monitoring the web pages. JRF will investigate the left over night and sort out as required. <<<< MJI suggested an extra step in checking the monthly master flats after having found a 10% gradient in the February Z-band illumination correction - albeit successfully taken care of with the detector zero-point adjustments. JRL and JRF will make it so for new monthly master flats. <<<< [e.g. Z_mosaic_std.fit in /data/apm44_b/vista/processed/20120220/] Rob Blake pointed out a batch of missing images for 20111124. Apart from comments on how can that be etc ..... it does appear the images are genuinely missing from the processed directory. JRF will regenerate them and everybody will try to think up weird scenarios about how this could have happened and what to do about minimising the odds of encore une fois. <<<< Some odd catalogue sequence aka catalogue files not from stacks on 20120110 triggered a few internal queries. Turns out the pipeline is not sentient but was aided and abetted by JRL. EGS investigated further and found out these were expressly taken for the purpose of allowing precision astrometry (sic!) from VISTA by mapping the field distortions. The fact that we have already done this and can readily generate more examples for free obviously passed the OPC by. VVV, Chile Branch, are not in our good books having "lost" virtually all of their online disk store, painfully shipped over the years on USB disks. Backups ? que ? Give or take a few files this is around 10TB of data which given their amazingly fast Internet access to here would take them to around the next century to copy over. EGS politely inquired if they had access to either an LTO4 or LTO5 tape unit since we can at least automate dumping a copy to tape. They have, and he also, politely, pointed out that then they would also have a BACKUP !! <<<< Shortly after Eammon Kerin inquired about getting a copy of all the Ks-band images for the variability study = ~3TB and exquisitely bad timing. A straight bat defensive prod back to the bowler was employed. At the end of the meeting JPE produced a bunch of VISTA Z filter curves out of his hat to demonstrate the 2 sets of long wavelength cut offs cunningly deployed. VISTA Detectors 1-4 & 13-16 have Z filters that cut off about 5nm longer than do detectors 5-12. In principle the colour equations 2MASS => Z_vista should be different for the two sets of 8 so the question posed was do observations show any discrepancies that could be attributed to this difference? Though this was taken as an action during the meeting, it didn't take long to answer and is appended here. The two sets of slightly different bandpasses probably explains a consistent small zero-point offset between the "inner" and "outer" detectors of about 0.02 mag on average. However, other than that there is no evidence of significantly different (i.e. measureably so) colour equations. Of course, the zero-point difference is already dealt with by the monthly detector-level zeropoint adjustments. 8. VST update -------------- This first part of the VTS update is taken verbatim from the CASU contribution to a report that the VST PI's are submitting shortly to ESO. "From February 29th Internet transfers of VST data from Paranal to Garching have been using the EVALSO link and VST data is now available in the ESO archive for CASU to transfer over the Internet some 24 hours later. CASU had previously already automated the transfer process from the ESO archive for available VST data and now routinely acquire VST data for checks and local ingest within a few days of the observations. The VST web pages at CASU for processing status (http://casu.ast.cam.ac.uk/vst/data-processing/) and survey progress and access (http://casu.ast.cam.ac.uk/vstsp/) have been upgraded to reflect the full status of what is happening. An internal CASU review of the first-pass VST processing at the beginning of January showed up several obvious pipeline issues that needed fixing in addition to highlighting several genuine problems with VST data. The latest pipeline version (v0.9) addresses all of the main image processing issues and all the VST data has now been reprocessed with this version prior to release to PIs. Work is still ongoing to fully address the intrinsic VST-specific problems but with promising results. In addition to the usual VDFS QC monitoring of average stellar seeing and ellipticity, and sky surface brightness and noise properties, CASU have also initiated a more detailed analysis of the image properties based on inter-detector comparisons. The well-aligned coplanar detector array coupled with the curved focal surface is extremely sensitive to imperfections in focus which are relatively easy to detect using the detector-level average seeing measurement variation available for each of the 32 detectors. Likewise the variation in average stellar ellipticity from each detector over the field is used to monitor rotator angle tracking problems. CASU are still waiting for a practical method of transferring the OB grades from ESO so that they can be incorporated in the data product files and also the progress database. After some experimentation, and verification, CASU are using the APASS (v5) all-sky photometric (http://www.aavso.org/apass) g,r,i catalogues to monitor both overall throughput and provide a working solution to the severe illumination correction problem inherent in VST data. A solution has been developed over the last few months to correct the ~0.25 magnitude systematic variation in sensitivity over the array due to scattering from within the camera and from the telescope and dome structures. The scattered light corrupts the flatfields, both dome and twilight, with the detail varying with external conditions. In turn this leads to spatially-dependent magnitude offsets made up of several inter-related components with different spatial scales and symmetries. Tests so far indicate that a correction to the level of +/-1% is achievable for the broadband u,g,r,i,z filters, and at the +/-2% level for the NB_659 filter after excising a full field central 2 arcmin wide cross in the case of the latter. As the fix is so complicated this involves producing a "fixed" version of all the relevant catalogues whereby all the internal flux values per detected object are corrected. After recalibrating these will form the basis for the standard user end products. The original catalogues will also be available, mainly to simplify implementing future improvements to the scattered light correction. A further severe and unexpected issue are the gain variations in several detectors. These occur randomly but most noticeably in detector #82 which shows excursions of up to 1/2 magnitude. The VPHAS observing strategy, of single exposures with large offsets to the companion pointing within an OB, highlights these particularly well. Ironically it also enables a possible solution to the problem at the catalogue band-merging stage Despite the illumination correction problem, even using only the 3 Standard Areas observed each night from the primary calibration plan, CASU are finding virtually all nights yield a very steady ZP measurement. ZP variations per pointing are generally already well within 5%. After fixing the illumination correction, by additionally taking advantage of the all-sky APASS catalogues, achieving a uniform global calibration at the 2% level looks feasible." There was a slight faux pas in transferring images (i.e. missed some) during the changeover on the 29th February, but this appears to have been a one off and was fixed for posterity with a slight modification to the transfer timing. The missing images were copied over, processed and added to the pot for that night. In addition to this, ATLAS, as usual, had a 2dF run lined up in April that required access to processed NGC data observed up to the end of March !! AKY and MJI shovelled the data through as fast as it would go and ...... as its all gone quiet assume it made it in time. As noted in the previous minutes, the U-band needs a fix to cope with its rubbish throughput and the shortly arriving ESO OB grades still haven't ..... arrived ...... that is. MJI also asked JRL to ensure that no VPHAS science frames are being used in creating fringe frames for the i-band fringe corrections. <<<< [due to the excessive image crowding in the Galactic Plane/Bulge survey] MJI had spotted a useful analysis of Paranal extinction published by Patat et al. 2011 and has updated the VST default extinction measures accordingly. In addition to these mods, v1.0 still awaits small changes to the colour equations and the odd tweak to the illumination correction software before it is rolled out. 9. Spectroscopic stuff ----------------------- Many meetings and telecons (GaiaESO, 4MOST, WEAVE) as noted earlier and steady progress on the GaiaESO processing, data formats and ICDs. JRL has been shovelling GaiaESO data through the CASU and ESO pipelines, finding the ESO pipeline foibles, and comparing results (and formats) with the Keele pipeline. EGS has been burning the midnight oil analysing simulated spectra for 4MOST and trying to shoehorn requirements, FOMs, and science goals into a coherent story. 10. ESO in-kind stuff --------------------- apm48 became the host for the HAWKI and VIMOS image processing project and now holds a full set of HAWKI data and a large chunk of the VIMOS data - though more of the latter is still winging its way here on ancient USB bricks. CPL version 6.0 est arrivee and will initially require a host of small changes to the already delivered VISTA software just to stand still. 11. AOB/news ------------ No time for juicy stories from CASUhelps. The rubbish opening/closing mechanisms for the upper south-facing windows in some of the APM offices were surprisingly fixed (at least for now). Any further failures will equal summary dismissal of same. The blinds for the upper windows are apparently on their way and will be installed shortly even though the yellow orb has been noticeable by its complete and utter absence this last month. MR boosted the productivity of the group by bringing an Expresso machine back from Italy, while Greg (Gaia) is improving the average girth of CASU by trying out assorted bread, cake, and dessert recipes on us. Meanwhile Beers of Europe continue to astound with their diversity - hic ! Continuing actions ------------------ JRL - investigate benefits of a "rolling" master linearity correction or even an average channel one STH - progress evermore the VISTA calibration papers writeups and circulate draft papers before the next CASU meeting JPE - finish off investigation of VISTA diffraction spikes by defining JRF affected radius -v- magnitude SC - keep chasing up acquisition of a new G-bit switch New actions ----------- SC - set up apm47 and apm49 SC - sort out ancient login accounts MJI SC - monitor and attempt fixes for sporadic Blade I/O problems SC - install X11 libraries and ds9 on apm45 ALL - register if interested in attending UKIDSS final workshop EGS - rationalise serving up of UVEX data MJI MJI - finish off March and April WFCAM monthly calibration updates MJI - organise av conf column updates for WFCAM standdown period JRF - sort out VISTA night of 20120215 JRL - add in extra check for master monthly flats JRF JRF - regenerate missing VISTA images for 20111124 JRL EGS - dump VVV data to pile of LTO4's JRL - ensure no VPHAS i-band images make it into fringe frames