CASU meeting Wednesday 31st October 11:30-13:00 APM Present: MJI, SC, AKY, STH, EGS, JPE Apologies: MR, JRL, RGM, NAW Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- JPE - have characterised the "radial" extent of diffraction spikes for STH VISTA the information will go in the technical web pages - ongoing <<<< MJI - cf. UCAC4 and PPMXL as alternative astrometric catalogues - ongoing <<<< MJI - updated bandmerging prog for VPHAS data - ongoing slowly <<<< SC - is still expediting the handover of the spare LTO4 library unit plus associated 30 TB disk system - progressing but slowly <<<< EGS - added UHS progress web page to the WFCAM survey monitoring pages MJI - still hasn't plonked down random apertures to measure tile noise <<<< STH - turned up in force for the IoA Computing Open Meeting on 2nd October SC and reported no major surprises affecting CASU MJI - continues to propagate UVEX band-merged products a toute vitesse <<<< MJI - finished off the August WFCAM zpt updates and released the data MR MJI - still hasn't put note on web pages about pawsky limitations <<<< MJI - though he did generate the VISTA zero-point updates for August JRL - whinged as usual about twilight flats at the VISTA IOT meeting EGS - flagged PI data locations on the VISTASP summary pages ?? STH - is still updating the ZP offsets for the VISTA calibration paper <<<< MJI - forwarded emails about access to commissioning data for NGC253 to JPE 2. WFAU minutes --------------- Much is quiet on the Northern front apart from a few hardware problems. It was noted that the VISTA Science Archive paper has been accepted by A&A which prompted some paper writing volunteering. 3. Hardware update ------------------- The donation of the LTO4 tape libray unit and the 30TB disk is becoming ever more urgent due to lack of disk space. Temporary workarounds have been instigated to circumvent the problem. The new fibre channel RAID array est arrivee hier matin. Now it requires some TLC and a bit of reformatting offline prior to installation. <<<< There then followed an interesting (worrying!) discussion on modern linux operating systems and the system-level appropriation of the lower number GIDs and UIDs. This will undoubtedly cause some grief further downstream to the more ancient members of CASU who, naturally, have low number UIDs and GIDs. For now though we decided on a GOP approach and stuck our heads in the sand to ignore it. SC raised the issue of retiring our Sybase server (apm5) when JRL gets a chance to migrate things to our new Postgres server - included as an action item to remind us to keep an eye on progress. <<<< 4. Meetings and telecons ------------------------- STH and SC attended the IoA Computing Open Meeting on 2nd October as CASU representatives and as noted earlier reported no relevant issues. MJI, EGS and JPE attended the ESO surveys meeting October 15-18th in Garching and reported back no major concerns but a clear need to finish off the VISTA calibration papers and publish the VISTA-Vega conversion offsets aussitot que possible. There was yet another VMC skypecon on 22nd October at which mr. reported on the positive feedback from the PSP meeting. Interestingly, the VMC survey is the least complete of the VISTA surveys and its expected total duration is now around 9 years. The VISTA IOT also on 22nd October was its usual memorable self with much talk about the VISTA in-kind pipeline upgrades and not enough on operational issues. One curiosity was a reported sporadic shift of 8 pixels of the entire detector readout for chip #8. We eagerly await some examples to investigate further. Upcoming meetings include: Science with 4MOST meeting 13-15th November; WEAVE System Managers meeting La Palma 21-22nd November; KiDS progress meeting Leiden 12-13th December; VISTA - a celebration ROE 17-18th January; and a Gaia-ESO survey all-hands meeting Nice 8-11th April. Some/all of which may/may-not be attended by a select/dissolute band of CASU brethen. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA, VST ------------------------ A quiet month for a change with only UVEX poking its head above the processing parapet and a request from Jonathan Zwart to process an INT WFC run. The latter was duly done and the former prodded de temps en temps. 6. WFCAM update ---------------- The odd passwd glitch interrupted transfers but otherwise a mostly smooth operation this last month. There were some apparently odd out-of-focus images for U/UKIDSS/B1 that got thrown away at QC checking stage on the nights of 20120928,20120929,20121001 without us realising at the time that these were deliberately taken this way. Once alerted to this we stopped wholesale removal of the processed products for them and will no doubt be regenerating the missing groups in the near future. [Note to us:- read weekly WFCAM summary more closely and note to JAC:- a reminder in weekly summaries for cases like this would be useful] The zero-point updates for August and September were done and the processed data products for these two months released (as were October's while these minutes were being typed up). Both the WFCAM raw data disk apm24_f and the processed data disk apm44_d are now full so we are switching temporarily to apm47_a while awaiting the installation of the new fibre channel system (actually a replacement for an old one with only one functioning power supply and no spares on the planet). There was one strange screwed up set of 3x3 interleaves with clearly the wrong offset values, part of a UDS stacked super frame sequence otherwise known as w20120930_00831_sf_st.fit. As this was the first sub-group the workaround was to reprocess sans the first chunk of 9 frames. MJI produced some CASU costings for background to the UKIRT sales prospectus and compiled and sent in the traditional 6-monthly UKIRT Board report. There were a series of interesting UHS consortium email exchanges regarding fast flat file access for UHS processed data. This would be no different to CASU provision of similar access done in the past for LAS and regularly done currently for many PI programmes, but don't expect us to call. Talking of which there were the usual early data delivery queries from assorted folk which are dealt with as they come in. 7. VISTA update ---------------- Raw data transfer, ingest and tape backups of same running smoothly and first-pass processing for September is completed. July and August processed data were finshed off and released mid-October. The Galactic Bulge season should be completed by now so the painful Ks-band variability data cataloguing should no longer dominate proceedings. JPE pointed out that the VIRCAM close cycle cooler will be receiving some TLC from December 8th to December 17th 2012 (noon to noon) so clearly, there or thereabouts is a good bet for the CASU Christmas party. Henry Joy reported some corrupt transferred UltraVISTA files which turned out to be a subtlely in the way he was transferring the data. EGS made the list transfer process even more robust to circumvent users being able to corrupt the file transfers. MJI spotted the lack of a P90 folder and asked AKY to ensure one gets created to keep eager web browsing users happy. <<<< The vast pile of LTO4 tapes for recent VVV data were duly delivered to Dante Minniti by EGS at the ESO Survey meeting. We look forward with eager anticipation to our reward. The bulge in the number of files and the size of the Bulge catalogues has eaten up disk space at two-three times the usual average so there was a bit of scramble to find some temporary storage space while awaiting the everlasting transfer of the 30TB disk. Whilst generating the zero-point offsets for August MJI noticed a bit of a gradient in the H flat for August 2012. As there was very little H-band data this month, and as much of it was twilight standards, and as the zero-point correction fixes the problem, there seems no point in generating a new H-band flat and reprocessing. However, it is a timely reminder to check the monthly flats for such effects. One useless night, 20120809, was reflagged as unprocessed after discovering the whole night was lost due to a technical problem with M2 - even though a tranche of science data was taken (which was why it got processed in the first place). [There is a residual action to fix the mysterious leftover v1.2 for that night.] <<<< RGM had previously raised an issue regarding the occasional blank OB qc status keyword for VISTA. EGS did a quick trawl and found 66 tiles with blank status in VHS data from 2011 & 2012. It seems that ESO are supplying us either with incomplete information at times, or the information supplied is incorrect which causes the parsing script to fail e.g. wrong end time and date for an OB. An example of the latter is below 2012-08-24 VIS 179.A-2010(F) 812323 0.00 A [P89A_Str06_ATL_RA_313_000_1_1_46]
(2012-08-25T08:58:33 to 1970-01-01T00:00:00) We should flag these cases to ESO, since it looks like for VST we have many more missing grades, and include everything in one email. <<<< There was recent query from Mike Read re: v20100809_00456_st_tl.fit a VVV H-band tile with a strange catalogue pattern that showed up in J,H or H,Ks CMDs/spatial distributions. The problem was almost undoubtedly caused by one of the H-band stacked pawprints, at the time, having a strange magnitude zero-point relative to the other five since the original tile catalogue did not have this problem. The poor old grouter did what it says on the tin and equalised the throughputs, among other things, and thereby introduced a 1-2 magnitude offset where this pawprint hit the tile. The current on-line pawprints here all had sensible magnitude zero-points. So, unsurprisingly a test regrouting from the original tile catalogue showed the problem had long gone and had presumably been fixed by an upgrade to the photometric calibration software around a year ago [tiles were not regrouted after the zero-point updates]. This field has an average Schlegel E(B-V) ~ 7.5 and has highly variable and patchy extinction regions which the upgrades to the calibration software were designed to cope better with. The new version of the grouted tile catalogue was made available to WFAU, but this is unlikely to be the only example in the Galactic Plane where this has happened. The global photometric recalibration of all VISTA data is still on hold until the photometry paper, and hence the colour equations, are signed off. Meanwhile more test examples are welcome. STH is working on the offsets between the Vega and VISTA photometric systems for all bands using a mixture of 2MASS, SDSS and WFCAM data for comparisons. He will report back on the results at the next CASU meeting. And the final VISTA item is an amuse bouche: it seems, unbeknownest to us, that since 20120301 we have been labelling VISTA processed data with CASUVERS= 'omegacam version 1.2' due to accidental use of a script shared with VST for inserting this single header card in the PHU. The only header item affected is this one card. The processing version i.e. version 1.2 is correct, as is everything else. We have now updated the relevant headers as above for VISTA data from 20120301. It affects all VISTA FITS images from this date (and catalogues too). Thanks to Bo Milvang-Jensen for puzzling over this and letting us know. 8. VST update -------------- Raw data transfers, ingest, and processing are all on track. The VST September processing was finished post haste due to Tom and his observing runs, again ! The illumination corrections for September were computed and the fixed catalogues made available through the usual route. VST too was getting a bit short of processed data space, but then a quick tally of where all the space had gone revealed that 25% of it was being used to store repeated duplicates of the master monthly calibration frames for each processed night - d'er! A quick relink alternative freed up the redundant space and gives us breathing space encore une fois. In addition to the traditional Autumn rush we also generated 1570 list-driven stacked u-band catalogues (both ATLAS and Chilean PI data) based on the best g-band data available for each suitable field. These also, of course, required the illumination correction step. The increase in depth, particularly for UVX quasar selection, is impressive, and the sleight-of-hand re: stacking the list-driven catalogue results (rather than the ATLAS and Chilean u-band images) worked a treat. The one VST foible still not yet addressed, the random gain changes in some of the detectors is next on the list. However, we did learn (needs checking) <<<< at the ESO meeting that the other half of the recalcitrant row viz detectors #85-88, the other worst culprits, have had their video board replaced too. Assorted contributions to the VST Survey Progress reports for the P91 OPC meeting were also generated and lobbed over to the ATLAS and VPHAS PIs. Mike Read (again) trawling through the many VST oddities (which for his pains resulted in us sending him all the nightly READMEs which explain many of them) spotted a default value (0.25) being set occasionally for the detector-level ellipticities recorded in the catalogues. This also happens with the individual detector seeing estimates. The latter are set to more obvious values (-1.0) and maybe an equivalent rejig of the default ellipticity values is in order. [The default values are triggered if there are not enough suitable images on the frame to generate a reliable estimate, not an uncommon occurrence for some of the U data in particular.] A future upgrade is planned for the VST classifier that should fix most of these foibles anyway and also make it more reliable for low numbers of objects on frames. A question from the VPHAS PI indicated the current VST photometric calibration method was not clear (true) and the reply is paraphrased below to remind us to put this information on the web pages. <<<< APASS is not being used yet for calibration of zero-points. VST is currently calibrated directly on Vega based on the few standard fields observed each (mostly) night. The calibration is currently in a VST Vega-like system, but as the u-band part of the Vega spectrum, and also the average standard star, and the detectors, are plummetting rapidly here it would be surprising if there were not offsets (and nonlinearities and duplicities) in the u-band calibration. With a stable illumination correction system in place, the next step is to redo all the zero-points based on the fixed catalogues - they are still based on the originals. At the same time we will compute an APASS ABmag zero-point and have that in the header too. When the recalibration is done we will compute a VST -> Vega offset for all bands and post it on our web pages. The other twist to follow up came from the news relayed at the ESO survey meeting regarding the supposedly superior properties of the dome flats cf. to twilights - hard to believe but needs following up with a few tests. <<<< Au l'autre main we have no plans to change how we process the data since it clearly works fine. The multi-$xx question -> is there both a stable dome illumination system and one where there is no variable outside light leaking in ? To be continued ...... 9. Spectroscopic stuff ----------------------- Gaia-ESO the Gaia-ESO pipeline products velocity tests look good with sub-km/s accuracy. There has been a lot of progress towards the 1st quarterly (v1.0) release of data including both Giraffe and UVES spectra, stacked as appropriate, and with a range of user-selectable download options (and this despite the UVES folk delivering non-standard format files). This 1st quartley release has now happened, despite the format issues, and Clare Worley begins work on the IoA's Gaia-ESO team on the 1st November and will take charge of QC matters and much of the user-side interaction - thank goodness ! These are supposed to be formal version-controlled internal data releases and we anticipate these releases will happen 3-4 times per year (almost quarterly then), with the second anticipated early ~February 2013. The requirement for these data releases is that they be 1) version controlled, so all later work is based on fixed information; 2) complete, in that all associated information is provided, including RVs, photometry, etc; 3) extensively quality-control checked before release, including specifically consistency of information across different wavelength settings, with the QC data also released. All work associated with UVES will continue to be done in Arcetri, and transferred complete to the CASU archive. All Giraffe-related work will be done in the UK, involving CASU and Keele. At some point we (CASU) need to do a detailed comparison with the ESO Giraffe pipeline - as described in its user manual at ftp://ftp.eso.org/pub/dfs/pipelines/giraffe/giraf-manual-2.9.pdf - and make a list of the modifications that we have made for the CASU version. <<<< 4MOST During the less interesting ESO survey talks MJI did a brain dump of technical issues required to be addressed to assuage ESO concerns over assorted sundry items such as sky subtraction, scattered light, cross-talk, Cramer-Rao bounds usw. This was forward to the project office to incorporate into their next submission. Meanwhile NAW and EGS either fended off some of the more overt excesses of the various simulation rounds or fed back assorted results of simulations to keep our German colleagues happy. A slightly-out-of-the-blue request for a 4MOST ETC plopped in our laps (teaching us to keep our mouths firmly clenched next time) and is being dealt with accordingly. WEAVE Fortunately all is much quieter on the WEAVE front - we just hope the synergy argument keeps on working. 10. ESO in-kind stuff --------------------- The quarterly report was duly written and delivered together with JRL's updated design document for the VISTA pipeline enhancements. A tarball of the progess with the enhancement software to date was also dispatched together with a hatchet job on what is wrong with the ESO-supplied HawkI pipeline and what we intend to do about it. VIMOS is next for the chop. AOB --- CASU job interviews took place this week to ensure no dull moments. We are ever hopeful. While JRL made the most of the UK equivalent++ to Medicare and we all wish him a speedy recovery. Continuing actions ------------------ JPE - put information regarding the radial extent of diffraction spikes for a STH range of values of pawprint NDIT*DITs on web pages MJI - compare UCAC4 and PPMXL as alternative astrometric catalogues MJI - update bandmerging prog to cope with weird VPHAS observing pattern SC - expedite the handover of the spare LTO4 library unit plus associated 30 TB disk system MJI - plonk down randomly placed apertures to estimate VISTA tile noise MJI - continue to propagate UVEX band-merged products to web page access MJI - add note to web pages regarding pawsky limitations STH - update the ZP offsets for the VISTA calibration paper for next meeting New actions ----------- MJI - reformat RAID array and coerce it into rack and thence connect it JRL - report on progess with retiring our apm5 Sybase server AKY - ensure a P90 folder gets created for VISTA processing monitoring EGS - fix mysterious leftover v1.2 manifestation for night of 20120809 EGS - compose list of ESO OB status blanks for VISTA and VST and send to ESO MJI - check if the video board for #85-88 really did get replaced MJI - add a note about current VST photometric calibration to web pages MJI - are the VST dome flats that stable and how do they cf. to twilights JRL - write up notes of limitations/mods needed for the ESO Giraffe pipeline