CASU meeting Wednesday 3rd April 11:30-13:00 APM Present: MJI, AKY, STH, JRL, CGF, GMX Apologies: MR, EGS, NAW, RGM, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news The March meeting was rescheduled by one week to christen the arrival and welcome the two new members of CASU, Greg Madsen (GMX) and Carlos Gonzalez-Fernandez (CGF). The extra agenda item aka progress on ALL papers was noted but discussion deferred until the next meeting when the major authors would all be present. 1. Actions from last meeting ----------------------------- STH - has the information ready to impart re: "radial" extent of CGF diffraction spikes, imparting to web world will be part of a general overview/update of technical pages contents by CGF <<<< EGS - contacted VPHAS PI regarding format for band-merged catalogues and as they are happy with their lot so are we SC - apm50 disk+LT04 handover almost there <<<< MJI - finally plonked down the random apertures on tiles-v-stacked pawprints to measure noise properties and got what it says on the tin STH - showed the results from a WFCAM comparison to derive the Y-band zero-point offset for VISTA. Z-band still requires some work <<<< JRL - have made some progress in replacing apm5 Sybase server by moving EGS some of the smaller DBs but the rest still ongoing <<<< MJI - the VST dome flats -v- twilight flats comparison is still on hold <<<< JRL - has produced some notes on the ESO giraffe pipeline foibles EGS - has booted the v1.3 upgrade and release out of the door after MJI found that 2012 was fine as is bar the odd ESO grade update ALL - cleared up home disk space usage with a bit of help and some old home accounts were migrated SC - was tasked with investigating the 5-10x slower NFS push-v-pull issues on apm44-49 MJI - checked during the meeting if CGF went to the 4th VVV meeting - nope AKY - is applying TLC to VISTA raw data offline backups - ongoing <<<< MJI - nada pasa con el V5 links <<<< STH - investigated the shuffled red star tests for the NB118 VISTA filter with not unexpected inconclusive results. He will investigate if any suitable VVV field test data has been taken. Another possiblity <<<< is to investigate if enough NB118 data on different fields exists to check detector-level corrections <<<< EGS - the tape copying has been pending awaiting the finish of v1.3 <<<< ALL - involved did check the changes list for the VISTA v1.3 release MJI - progressed the progress on papers agenda item -> next meeting <<<< 2. WFAU minutes --------------- No significant comments. 3. Hardware update (and software) ------------------ There were a few queries about casutools use, particularly for VST data. These were closed bar two which highlight upcoming issues relating to list-driven photometry (e.g. aperture corrections, morphological classification) and illumination corrections (e.g. howto implement). <<<< There was also a general query related to recent updates in casutools, the list of which should be more easily visible on the web pages to save unnecessary downloads. CGF was volunteered take charge of these technical web page updates and to help EGS with general maintenance and updates for the plone web pages. <<<< The WFAU apm7 uklight and wfcam access entry point was resurrected thanks to SC and should all now be back on the road. Some useful software tools such as xv, fv, fverify etc... seem to be absent from some of the more recent apm computer acquisitions. A brief list of must-haves needs drawing up and propagating across the relevant machines. <<<< One of the recently acquired fibre channel arrays was installed on apm36 for time series data storage. The other was installed on apm47 to be used for storage of vst/orig files (i.e. earlier raw data) freeing up space on apm45 for more recent raw VST data storage and ongoing transfers from ESO. STH spotted a possible bug in imcore_list where it looks like (at least for cattype=2) the RA,DEC columns returned in the list fits file are in degrees while the column description says radians. (The columns should be radians for consistency across catalogue products.) MJI commented that 3-D data visualisation was advertised in DS9 version 7 but as he's struggling to cope with 2-D would forego the experience. SC reported on proposed University of Cambridge changes to Janet network traffic charges to the departments and colleges. The short summary is pro-rata costs per Institute up to a 5% cap. Most of the heavy traffic from here goes via UKLight and although there are some changes for that also planned they should not affect the cost significantly. CGF and GMX were reminded to decide what sort of desktop hardware they wanted bearing in mind the pros and cons of the various options available. 4. Meetings and telecons ------------------------- The usual VMC monthly telecon was superceded by the upcoming Potsdam meeting 9th-11th April 2013. JPE is attending and will report back on any relevant issues at the next CASU meeting. <<<< MJI batted for 4MOST at the ESO review of the Phase A studies submitted by the 4MOST and MOONS consortia 4th-6th March. The panel recommendations should be known by the end of March and these will be forwarded to STC to make the final decision supposedly by the end of April. WEAVE underwent its PDR which highlighted several issues not the least of which was Data Management and CASU's role, currently under renegotiation. The 4th VVV Meeting happened on March 21-23 in Vina del Mar. Ever optimistic we assume that silence is golden. JRL, EGS and NAW are off to the Gaia-ESO survey all-hands meeting in Nice 8-11 April and will report back at the next CASU meeting. NAW will do his usual multi-tasking and also attend the CHEOPS meeting in Geneva on the 11th April and present our quick-look pipelines plans. The other upcoming meetings of note are the Euclid OU-VIS meeting in Paris 22nd-23rd April for which MJI has written an astrometry howto in lieu of attending. NAW and EGS? will fly the flag. The other being the ESO inkind mid-term review in Garching on the 30th April for which JRL, EGS, AKY, MJI and GMX all drew the short straws but could at least rustle up a five-a-side team. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ All running smoothly as far we know. An oldie but goldie about strange INT WFC WCS parameters in some early IPHAS data popped up. This turned out to be a bunch of surplus extra garbage terms that are normally expunged from the headers before processing begins. Presumably some of this cleaning step had been accidently omitted and someone is using a particuarly picky bit of software. All of the band-merging backlog of processed UVEX/IPHAS data has finally been cleared and the files made available to the PIs. 6. WFCAM update ---------------- Transfer and processing proceeding on track bar the odd transfer hiccup due to our home directory disk at the JAC end going on strike and camera#4 losing the first half of its data conversion on 20130331 thereby requiring a gentle nudge from both ends. Release of January and February data awaits more moments of respite and indeed the computation of the illumination corrections. In producing the March master flats some extra "muck" on the field lens/cryostat window put in an appearance. This seems to have been deposited during/caused by the winter storm of the last week of February. It shows up as mainly low level out-of-focus absorption, in the same place for Z,Y,J,H when comparing with earlier master flats and as out-of-focus low level emission in the same place in K. It was only visible over a region covering detector#3. It is low level and processes out so it wasn't a major issue. After a bit more checking of processed data, that region was clean on the last night before the bad weather i.e. 20130220 and was present the first night after the bad weather that we processed i.e. 20130227 where it shows up clearly in one of the derived J-band sky frames, together with the usual parquet floor pattern. According to JAC this was most likely due to ice patches on the cryostat window. The field lens was CO2 cleaned 20130228 and the detritus continued to show up after that. An unresolved issue with apparently large numbers (~20%) of images flagged morphologically as -7 in the UKIDSS 2nd epoch data (one or more bad pixels in aper3) was raised by Phil Lucas. It is not yet clear how widespread this might be or even if it represents a problem. It will be left to simmer until more al dente. 7. VISTA update ---------------- Raw data transfers, ingest, tape backups (nearly), processing and releases proceed apace with January being released and February not far behind. JPE appears to have placated ESO re: their query regarding the VISTA ETC web pages. Let's hope it stays that way. VISTA Quality Control of observations at ESO has undergone some changes in an attempt to improve the overall observing efficiency regarding repeating OBs. The obvious issue here is partial tile OBs for VVV where having the same pair of tiles used for sky estimation is a requirement. It will be interesting to follow the discussion about compensation time for these new ESO OB grading rules. Although the v1.3 release has finally happened progress with it wasn't helped by a weird file permissions problem due to extended ACL masks being set by default on the ZFS file systems on apm29 and apm35 - ignorance is definitely bliss on this one. There was (to us at least) an amusing phone call from an old pal in the detector division at ESO regarding their recent discovery, during some instrument tests, of stripes due to pickup on the VIRCAM detectors. Apparently nobody told them of this phenomena! So after enlightening them further we also pointed them to ESO's very own trend analysis web pages on this very topic. However, it did highlight one issue, we don't have a picture of this phenomena on the CASU techincal web pages whereas we do have a full report on the older IoA vdfs/ web entry point => time for a bit of fusion. Among the joys of the 4MOST review was a chance to talk to Thomas Seiffert who mentioned a couple of interesting operational items pertaining to VISTA and VST. Apparently in addition to the VISTA hot air dewer dew blasting in February, they also cleaned the M1 mirror in December so it will be interesting to update the ZP trend plot and see if we notice anything. TS reckoned <<<< there was a 5% improvement in the dome flats throughput in December [though a quick post-meeting inspection of zero-points by MJI and STH found no obvious changes.] See next section for the other gem. 8. VST update -------------- All the VST January and February processing is finished, and much of March too. The illumination corrections for January and February are done and await EGS ingest prior to release. Raw ingest is running about 4 days behind observations, plus ca change. The other gem ...... VST apparently has a bistable M2 support system which if restarted at any time during the night can cause an absolute pointing jump of around 30 arcsec. If it happens it is almost certainly accounted for in our astrometric calibration so normal checks wouldn't pick it up, but presumably one of delRA delDec WCS solution keywords would. So it would be worth plotting the derived pointing error -v- time and see if it shows up. <<<< Upcoming issues will include any P91 Chilean U data which will no doubt require new permissions setting. <<<< The VPHAS PI asked for a small change to the VST DQC 5-sigma limits reported in the QC database along the lines of min, max, med. <<<< APASS DR7 has been released and acquired by JRL and will be used for future illumination corrections. The VST Phase3 window of opportunity est arrivee and requires a tranche of ATLAS and VPHAS data to be delivered by the end of April. After a brief flurry of emails this delivery will be restricted to pawprints and associated catalogues given that all the science is currently being done only on these. <<<< Bundling up the VST illumination correction software in CASUrtools keeps returning and it seems there is enough external interest in using it to make it worth spending the effort to make it so. <<<< However, this will not be the end of the saga since trying to apply these corrections to images for alternative analyses opens up several other different cans of worms. Apparently VST has been subject to some baffling (and even confusing) experiments on several nights toward the end of March. Quite what the main conclusion, or even state, of these are remains unknown. MJI will contact the chosen one of the 3M's to attempt to find out. One thing <<<< though is certain, any flatfields taken on these nights will not be of much use. 9. Spectroscopic stuff ----------------------- Unbeknown to us, testing ESO-Reflex workflows for FLAMES/Giraffe and UVES and reporting back on issues seems to have been agreed by the Gaia-ESO PIs. The half-empty hat views this as an unwanted chore, the half-full hat as a possibly useful learning experience relevant for later ESO inkind deliverables. <<<< JRL has concocted a cunning plan that minimises the number of interpolations for the 1-D spectral products and will factor it into the "improvements" bundled up for the next release. He also finally, as noted earlier, made a list of the quirks of the supplied ESO Giraffe pipeline which begins with ..... "What's wrong with the ESO Giraffe pipeline -> it crashes regularly i.e. when you use it" ...... and proceeds apace ....... [the following page was expurgated - becoming a regular habit in these minutes] As expected the parametric analysis being done by the many Gaia-ESO groups not only yields a broad spread of results/no results/varying definitions of NULL values usw., but also reveals a complete inability to follow instructions regarding formats - plus ca change then! 10. ESO in-kind stuff --------------------- EGS has put together a web page showing the VIMOS & HAWKI processing status http://casu.ast.cam.ac.uk/internal/processing-status. GMX will be analysing the master bias, darks, and flatfield frames generated by the trial pipeline to assess assorted features to help define calibration file update frequency, assess noise and detector properties (levels, noise, gain ....) and identify any systematics quirks (vignetting, pickup noise ....), comparing flats produced by the current ESO pipeline with CASU version and so on. This will all be useful information/ammunition for the mid-term review. <<<< The review will be chaired by Martino Romaniello, head of the Back End Department at ESO [we did not make that one up] JRL has already delivered a pile of documentation plus software plus test data and AKY has been seconded to help out with overviewing reflexing, not to mention Keplering actors. 11. AOB ------- CASU folk will be giving the Wednesday lunchtime colloquia on the 24th April which will consist of an overview of the current data processing infrastructure and science data products for WFCAM, VISTA and VST. This will be followed by another of the legendary CASU tastings. Continuing actions ------------------ STH - impart information regarding the radial extent of VISTA diffraction spikes -> web pages SC - dot the i's and cross the t's on gaia01 -> apm50 STH - compute the ZP offsets for the VISTA calibration for z-band JRL - report on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights SC - investigate possible NFS foibles causing push-pull transfer asymmetries AKY - keep applying some TLC to the VISTA raw data tape backups MJI - que pasa con el v5 links for WFCAM EGS - sort out the VVV LTO4 tape transfer MJI - ensure progress on ALL papers is an agenda item at the next CASU meeting New actions ----------- STH - have any NB118 pointings been taken on VVV fields for v.red star tests MJI - does enough NB118 data exist in any one month to compute the usual detector-level corrections ALL - thought experiments required for the general external use of list-driven photometry and use of illumination corrections CGF - review general technical pages contents from a content and information finding perspective ALL - draw up a shortlist of must-have software utilities for apm45-49 JPE - report back on items of interest/relevance from VMC meeting EGS - update the zp trend plots for VISTA to check December and February and if so generate them EGS - check if QC DB allows an overview of delRA delDec offsets for VST MJI - contact VST PIs to enable access to Chilean P91 data EGS - review reported summary of QC params for VST in vstsp/ (+ WFCAM,VISTA) EGS - progress VST phase3 deliverables JRL - magic fitsio_vst_fix into casutools MJI - contact Monika to inquire about VST baffling experiments AKY - jump at chance of learning more about Reflex workflows aka Gaia-ESO GMX - generate some trend and analysis plots for calibration files for HAWKI and VIMOS for use in upcoming mid-term review