CASU meeting Wednesday 1st May 11:30-13:00 APM Present: MJI, AKY, STH, JRL, CGF, GMX, JPE, EGS Apologies: MR, NAW, RGM Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- STH - "radial" extent of diffraction spikes is being imparted to web world CGF together with a bunch of other desirable improvements <<<< SC - apm50 disk+LT04 handover happened whilst writing up these minutes ! STH - a Sgr photometric systematics query from an angst-ridden user MJI delivered a useful byproduct aka a beautiful two-colour Z,J,Ks diagram for 7 contiguous tiles centred on Sgr dwarf from which the Z-band Vega offset looked rather well-defined. Spurred on by this we will attempt to corroborate with suitable VVV first season Bulge data and revisit the Y-band offset too <<<< JRL - slow progress in replacing apm5 Sybase server but spluttering on <<<< EGS MJI - the VST dome flats -v- twilight flats comparison is still stalled <<<< SC - is still investigating asymmetries in NSF push-v-pull transfers <<<< AKY - is still applying TLC to VISTA raw data offline backups <<<< MJI - after a discussion with EGS decided it was simpler to remove EGS the version links to make consistent with other web pages, the overall version log page is readily accessible as an alternative STH - the red star tests actions have been shelved after a tome of a draft paper on characterising same was sent round by Bo Milvang-Jensen EGS - the tape copying will be resurrected shortly as v1.3 is completed <<<< MJI - agenda item: progress on ALL papers was included and after a brief discussion became an independent get together which will report back at the next CASU monthly meeting <<<< ALL - pondered and chewed over generic use of list-driven photometry and illumination correction issues, particularly for VST CGF - is reviewing general technical pages contents from a content and information finding perspective and will include more visible CASUtools update information as part of this - ongoing <<<< ALL - came up with short list of "must have" packages for apm44-49 and installation of these is being progressed JPE - told a brief story of items of interest/relevance from the Potsdam VMC meeting EGS - the ZP trend plots for VISTA to check December and February throughput was a wild goose chase as Thomas had clearly! meant VST for the December M1 scrub, but it got done anyway and is appended to the bottom of the photometry page under vista/technical. There is a small apparent loss of sensitivity over the last few months but whether or not this reflects a seasonal variation in extinction remains to be seen EGS - has not yet checked if QC DB allows an overview of delRA delDec M2 support offsets for VST <<<< MJI - contacted VST PIs to enable access to Chilean P91 data EGS - reviewed reported summary of QC params for VST in vstsp/ (+ WFCAM,VISTA) EGS - progressed the VST phase3 deliverables despite ESO JRL - still needs to magic fitsio_vst_fix into CASUtools as part of the slog to make illumination correction available off-site <<<< MJI - forgot to contact Monika about the baffling VST experiments before the meeting but did so whilst writing up the minutes. The short story is that unsurprisingly " ...... several possible improvements to the baffling have been identified. With the temporary baffles installed, stray light features in short exposure images were completely eliminated ...... common action is being taken by the engineering departments in Paranal and Garching, but precise dates and/or timescales are not known. Also blackening of some telescope parts is under discussion." AKY - jumped at chance of learning more about Reflex workflows aka Gaia-ESO GMX - generated many trend and analysis plots from calibration files for HAWKI and VIMOS for use in upcoming mid-term review 2. WFAU minutes --------------- The acquistion of a 150TB NAS server box was noted, as was copying ALL the VISTA data back up again ! Interestingly the transfer speed out of here on UKlight is almost bimodal at 10MB/s or 40MB/s - not exactly obvious why. We are curious about the cost/performance of such large NAS boxes. 3. Hardware update (and software) ------------------ The library LTO4 unit plus thumper est arrivee enfin and the disk array will be used for VISTA processed data storage from April onwards. MJI cadged a spare rack liberated as part of the pre-works for the IoA kitchen update scheduled for late summer. The new desktop hardware for GMX and CGF finally made it and is now holding down desks and/or warming feet appropriately. How close are we to being able to retire apm15 which holds vista and wfcam qc info ? MR and EGS will investigate. <<<< The imcore_list possible bug ended up missing from the actions from the last minutes. So ...... is repeated and updated here. Aka STH spotted a possible bug in the CASUtools imcore_list where it looks like (at least for cattype=2 i.e. WFCAM style) the RA,DEC columns returned in the list fits file are in degrees while the column description says radians. [The columns should be radians for consistency across catalogue products.] A quick check post-meeting revealed that this is indeed the case and also happens for cattype=6 i.e. VIRCAM/VST style. Its amazing isn't it how ......... <<<< 4. Meetings and telecons ------------------------- The March and April VMC monthly telecons must have been superceded by the Potsdam meeting 9th-11th April 2013 since all is quiet on the VMC front. Assorted CASU folk gave the IoA Wednesday seminars on 24th April and followed that up with an eclectic spirits and cocktails session to welcome GMX and CGF. Star turns were most definitively the Pisco sours, Dafydd's sloe gin, and the palate-cleansing guacamole. Both 4MOST and MOONS have been recommended to proceed apace (as also has WEAVE). Details of CASU's potential data managements roles in these spectropscopic survey facilities are under negotiation and looking promising. These will build on the experience and pipelines developed for the Gaia-ESO survey. NAW and EGS attended the EUCLID OU-VIS meeting in Paris 22nd-23rd April and presented the astometry plans plus an update on the time series alerts. JRL, EGS and NAW (plus other IoA'ers) attended the Gaia-ESO survey all-hands meeting in Nice 8th-11th April. Much useful feedback and exchange of ideas and informaton ocurred despite the large numbers attending. JRL will now polish off his mods for the deferred iDR2 and generate the real version as opposed to the rebadged iDR1.5 now clogging up the disks. JRL, EGS, AKY, MJI and GMX duly attended the ESO inkind mid-term review in Garching on the 30th April and the AirBrau pub in Munich airport shortly thereafter. Both operations seem to go down well. There is a upcoming VISTA IOT scheduled for May 28th where the main discussion item will probably be the impact of the new do-not-repeat OB rule. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ All continue to run smoothly and indeed all is otherwise quiet on the UVEX/IPHAS processing front. There was an interesting AF2 glass plates query for measuring metrology of the WHT top end related to tracking down astrometry issues for WYFFOS. All we need is a measuring machine. 6. WFCAM update ---------------- The usual smooth operation of transfers and ingest apart from the odd hiccup this time due to wfdr1 and wfdr5 being swapped. However, thanks to Maren's efforts we didn't even notice and the automatic scripts papered over the cracks without us knowing. January, February and March data was released mid-April and at the time of writing April data has all been processed and awaits its final photometry updates. There was a query from a UKIDSS user regarding a strange J-band magnitude (cf. to 2MASS) of a candidate distant halo C-star. The UKIDSS H and K-band photometry agreed well and it transpired that the J-band measure had been flagged in the classifier as noise-like. The second epoch J-band measure from the UHS was available and that was fine on all counts so mystery solved. The moral as usual is that searching for strange coloured objects in large area surveys is a good way of finding spurions as well. 7. VISTA update ---------------- Raw data transfers, ingest, tape backups (nearly), processing and releases proceed as usual with Febuary being released as these minutes are being finished off and March (with a lot more VVV data) is now not far behind. Thanks to JPE's webpage updates the VISTA ETC saga appears to have been laid to rest, well at least for now. The urge to find any VVV equivalent NB118 data seems to have vanished given the draft paper from Bo already noted. As it was not obvious this approach would yield anything useful it has quietly been forgotten about particularly since if detectors#2 and #3 aren't the culprits one of the Jims will eat the other's shorts. Following hot on the heels of the VISTA v1.3 release was feedback from ESO ASG on the summer 2012 phase3 deliveries. Apparently 9 months downstream they've noticed a few things amiss (maybe) so some of these deliveries are still not closed off ! Related to this was a query from WFAU about provenance and reference files Unbeknownst to us EGS had solved this issue some way back with a cunning plan: aka some surveys only send tiles, so the PROV keywords reference the single images (i.e. the original unprocessed files we get from ESO) with their ESO names; other surveys send tiles and pawprints, so the tiles PROV keywords reference the pawprints and the pawprints reference the original single images - cunning but not clear how much use for potential punters CASUhelp is IN: A query about externally available CASU pipeline software not doing a good job for use in photometric calibration of VISTA data threw us for a while, since there isn't any. It turned out some ancient prototyping code had a Groundhog Day experience and was being replenished on an old web page portal every 24 hours. It no longer is. A query from Iain McDonald about photometry systematics in PI data for seven contiguous tiles (Z,J,Ks) taken centred on the Sgr dwarf came in via Mike Read. To cut a long story short, the data is stonkingly good and calibrated well within spec. As a byproduct it even suggested an independent way of calibrating/checking the Z-band (and Y-band via suitable VVV Bulge data) Vega offsets (as noted earlier). VISTA observations of Stripe 82 (Jean-Paul Kneib) are now completed and have been dispatched to the PI. However, other impatient users abound and M. 091.C-0795(A) is already whingeing about the month delay in having his April data fully processed. Bored with answering questions about effective gain in tiles, CASUhelp updated the relevant technical web page with some brief stray-light-removing notes. VVV folk are on the ball and shortly after the release of v1.3 came back with a data status query for obsname='b236j', which now reads "Aborted (R)", "()" and "()" for H, Ks, J, cats respectively, whereas before it was OK grade-wise. As a double check the file from ESO from which this grade was derived was examined and indeed this is what it said. For no obvious reason ESO have changed the grade of this OB to R (repeat later - it certainly will be later as this data was taken in August 2010). As usual we always advise to check the data if in doubt since the ESO grades (or even regrades) are not 100% reliable. A question came in from Mike Read regarding problems with use of the VIRCAM illumination correction tables in fitsio_cat_list. It must be admitted that the code for the illumination correction option was only written with WFCAM in mind and has not been upgraded (different table sizes) for VISTA since the illumination corrections there are small and about the same size as the errors in estimating them. In addition to the different table dimensions unlike for WFCAM the computed xi,xn for use in the table also need to be derotated to apply correctly since they effectively refer to detector x-y space, albeit in xi,xn coordinates - capice ? CGF reported on a stacking error when processing 20130317 due to a bright star causing applied shifts to be rubbish - all thought we did a global computation of shifts for the array - que ? <<<< MJI reminded everyone of Richard Hook's ongoing request for colourful PR images so if you stumble across any suitables ones let him know. STH asked if JPE's PI Carina data might be suitable ...... and ????????? was the recorded answer. 8. VST update -------------- All the VST March processing is finished and the products for January through March have been released. The usual ATLAS band-merged and list-driven files have all been updated too in advance (for a change) preparation for upcoming AAT 2DF runs in June and July. During early Phase3 preparation Janet Drew noticed some missing VPHAS data. This rang a few alarm bells since we did occasionally used to have trouble with the ESO archive availability of certain files when we attempted to copy them over. Quite how or why files were mysteriously not available to us is another matter. At the time we spotted most of this and eased back on our copying lag, but clearly still managed to miss a few files. By stunningly good planning we always put dummy files in place for any that were unavailable to save us the problem of having to re-index the run number sequences we use for filenames if and when they ever became available later. Most of these files were missing in large blocks towards the end of the nights, typically around 100 files. It was easier to remove the odd few that did get through and (re)process them in blocks. For the record these (re)processed blocks are 20120428 186-256 20120518 183-209 20120705 80-169 20120713 122-197 20120716 118-288 Easing back the copying lag to 4-5 nights from September 2012 seems to have solved the problem and this list above (thanks to EGS) seems to be all the VST nights affected back to the start of operations. These have all now been reprocessed, ingested into the QC database and made available. [For the record, previous (re)processed nights were for missing Chilean data 20121013 200-205 20121025 130-139 20121112 155-168 ] We also spotted since the beginning of P91 that ESO have changed something at the ESO request system end which required a few mods to scripts to download data automatically - d'er ! Meanwhile the VST Phase3 saga rumbles on. A well-timed disruption of Phase3 services by ESO on Sat/Sun 27/28 Apr 2013 is a good laugh as they are still checking the VISTA products we sent last summer and they require the VST data to be there by 30th whilst they are down over the weekend - noch einmal d'er! In preparation for Phase3 EGS has been testing their validation tool and there are some errors which appear because of missing keywords in the headers. TL_ID - tile identifier, unique within the Public Survey TL_RA - RA of the centre of the ideal tile as defined usgin the SADT or alternative preparation tool TL_DEC - DEC of the centre of the tile TL_OFFAN - Rotator offset angle mainly related to the fact these are not tile products - encore une d'er! For VISTA these are added by SADT but they are not present for VST. Suitable bodges have been created by EGS to circumvent these foibles. Latest news on this is that we have filled up their disks with Phase3 products and they can't read any of the files, so delivery is once again paused while they sort things out - good grief^2. Apart from that everything is fine. 9. Spectroscopic stuff ----------------------- Not much new to relate other than after the Nice meeting and a few email iterations WFAU science archive are now getting the spectra. WEAVE, 4MOST and MOONS negotiations proceed apace. 10. ESO in-kind stuff --------------------- The ESO inkind mid-term review went well, what more need we say. One of upshots of that meeting was the need to assess the reliability, stability and non-linearity of the gain measures for HAWKI, GMX will investigate. <<<< 11. AOB ------- STH briefly reported on NGTS, plus CHEOPS and Echo, involvement. Meanwhile NASA went ahead and selected TESS with a provisional 2017 launch - clearly exoplanets rock. We had a very constructive meeting with Pete, Richard, Tom and Simon from Warwick and MJI, Aimee, Marion (Geneva->IoA) and STH on Friday 26th. Main discussion was how we (CASU) process time domain photometry. STH downloaded some of this season's data taken by Tom and Marion which has much improved FITS headers and at least one interesting transiting object which Marion caught. RGW wants us to send him a minimum list of FITS header keywords we want be present. <<<< There is an upcoming CHEOPS science meeting in Bern, 15-16 May, NAW can't make it so STH will probably go. Continuing actions ------------------ STH - impart information regarding the radial extent of VISTA diffraction spikes -> web pages CGF - review general technical pages contents from a content and information finding perspective, including CASUtools update information STH - compute the ZP offsets for the VISTA calibration for z-band and cf. with MJI cunning use of two-colour diagrams in Sgr and Bulge regions JRL - report on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights SC - investigate possible NFS foibles causing push-pull transfer asymmetries AKY - keep applying some TLC to the VISTA raw data tape backups EGS - sort out the VVV LTO4 tape transfer EGS - check if QC DB allows an overview of delRA delDec offsets for VST JRL - magic fitsio_vst_fix into CASUtools New actions ----------- MR - figure out how to progress retiring apm15 EGS JRL - multiply RA Dec columns produced by imcore_list by atan(1.0)/45.0 MJI - organise an off-line get together to discuss progress of papers CGF - investigate what caused the stacking error for night of 20130317 JRL GMX - generate some gain measures for HAWKI and investigate their foibles STH - forward an example FITS header to NGTS bods at Warwick or point them to the WFCAM astrometry page