CASU meeting Wednesday 27th February 11:30-13:00 APM Present: MJI, AKY, STH, EGS, RGM, JPE Apologies: JRL, NAW, MR, SC Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- STH - has the information ready to impart re: "radial" extent of diffraction spikes, imparting to web world is still ongoing <<<< EGS - contacted VPHAS PI regarding format for band-merged catalogues comparison with their "blue" and "red" equivalents is ongoing <<<< SC - handover of disk+LTO4 on hold until SC returns <<<< MJI - running out of ways to report lack of progress on plonking down random apertures to measure tile noise <<<< STH - is still working on computing zero-point offsets for VISTA Y and Z <<<< JRL - with help from EGS has made some progress in replacing apm5 Sybase EGS server. Our friends at CDS are moving our copy of VizieR to the apm49 postgres server even as we speak. Moving some of the smaller DBs will occur in the next week or so <<<< MJI - the VST dome flats -v- twilight flats comparison is still on hold <<<< JRL - has started writing some notes on the ESO giraffe pipeline foibles <<<< EGS - produced a wfcamsp/ equivalent to vistasp/ and vstsp/ and almost instantly received a request for most of the processed wfcam data said user was redirected to the WSA EGS - the v1.2 -> v1.3 tile catalogue part of the upgrade is completed for 2009-2011 leaving just the minor matter of relocating these files and updating assorted header items. <<<< MJI needs to check how far into 2012 to continue this pusch <<<< ALL - seem to be filling in their absences entries with renewed vigour ALL - organised who is attending what where why how and when meetings MJI - unblocked the WFCAM data release drain for November and December data MJI - updating the aperture corrections and corresponding zero-points for Feb 2012 to Nov 2012 WFCAM data inclusive to fix the accidental use of the wrong Moffat coefficient as part of computing a total flux estimate is now completed MJI - the 3 nights with extra VST ATLAS Chilean data have now been patched though the next update of list-driven and band-merged products awaits completion of February processing MJI - attempted to double check with the VST PIs regarding the Phase3 deliverables (see later VST notes) ALL - signed card for SC and some even helped deliver it 2. WFAU minutes --------------- No comments. 3. Hardware update ------------------- Almost as soon as MJI went off skiing the air conditioning in the smaller of the two computer rooms decided to join him on the slopes, thereby cooking several computers at Gas MK-I until the high temperature alarm went off. The subsequent failure of the cooked system disk on apm7 could not possibly have been related. Given its vintage, cloning another system disk required some creative thinking, and even a cunning plan. However, apm7's equivalent to mitochondrial DNA is not the same as before so we anticipate a few emails shortly. The Apache web server on apm3 was upgraded following the recent UCS friendly? probing. The last of the recently acquired fibre channel arrays was installed on apm48 and is being tested to destruction prior to actual use. MJI asked if ALL could clear up /home disk space in advance of the two <<<< new arrivals turning up on 1st April. STH volunteered to help shift off some of the old accounts that mysteriously are all related to studying transient phenomena, apart from themselves. <<<< The NFS asymmetry of pushing files -v- pulling them across has crystallised to the extent that its time to investigate. It will be bundled up for SC to investigate to stop her getting bored when she returns. <<<< 4. Meetings and telecons ------------------------- MJI missed the VMC telecon in January but did at least read the minutes. JPE did attend the telecon but there was nothing much relating to processing of note and as usual another VMC telecon is coming up tomorrow. NAW attended the EGAPS meeting on 4th-5th Feburary in Nijmegen, plus, with EGS, a subsequent telecon on the 25th February to discuss spectroscopic followup of interesting sources. At the Nijmegen meeting a request for easy access to postage stamps of interesting sources was raised. As it happens that facility was already in place and subsequent mods have since improved it to deal with candidates on industrial scales. Upcoming meetings still include: the 4th VVV Meeting 21st-23rd March in Vina del Mar - though unsure anybody from here is going (MJI will check if Carlos is going); <<<< a Gaia-ESO survey all-hands meeting 8th-11th April in Nice - EGS, JRL ++; a VMC meeting 9th-11th April 2013 in Potsdam - JPE? MJI?; and have now been enhanced with an NGTS meeting 26th March in Geneva - STH; and a Euclid OU-VIS meeting in Paris 22nd-23rd April - MJI? NAW? EGS?. In addition MJI will attend the 4MOST presentation to ESO 4th-5th March, with MOONS' turn being on the 6th March, and assorted CASU bods will also be at the Gaia CU5 Plenary knees-up in Cambridge 19th-21st April. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ The raw archives are all up to date and the off-lines backups are mostly in good shape. AKY will apply a bit of TLC to the VISTA offline backups to bring them up to date. <<<< Much of the backlog of UVEX/IPHAS processing - mainly belated band-merging for the former - has been cleared and the files made available to the PIs. The APM guide star catalogue conversion software saga rolls ever onward but we took pity on them and sent them a linux-compatible version. A query about iphas finders not working revealed another alternative route we were not aware of hosted in the postgres collection. This was fixed. 6. WFCAM update ---------------- Due to some problems with the acquisition computer of camera #3, WFCAM operated on the nights of UT 20130203,04 with only cameras #1,2 and 4 collecting data. Assorted choice programes aka CMP/2, 13A/K2, 13A/H28B made good use of the time. A server problem for camera#1 at JAC on 20130131 went unnoticed at our end with the night being inadvertently flagged as "no data". The weekly summaries came in handy for spotting this one and the issues were quickly rectified with Maren's help at the JAC end. Despite these few wake up calls, receipt of data, ingest and processing is on track. As noted earlier, November and December data have finally been finished off and January awaits a moment of respite. A couple of interesting emails circulated summarising the current state-of-play for continuing operation of UKIRT. Apparently UKIRT was the most productive telescope on the planet last year and WFCAM produced the most papers of any instrument on a ground-based telescope. The usual suspects wanted fast access to processed data and there was even a query about getting access to the illumination correction tables. It turned out the web link for these was almost, but not quite, correct, it should be http://casu.ast.cam.ac.uk/wfcamillum/ Apparently the version number link ne marche pas tout le temps, but at least the overall version log link does. <<<< 7. VISTA update ---------------- Raw data transfers, ingest, tape backups (nearly), processing and releases proceed apace with November and December having left the building. January has also gone through phase-I of the pipeline and awaits its monthly photometry correction before phase-II processing can begin. There was apparently dew build up on the camera window around 6th February which was "blasted off" with a hot air blower held at blue-touch-paper length. We await delta-zeropoint analysis with interest ? The long-forgotten VISTA ETC saga re-appeared with a brief flurry of emails about the availability of the CASU version. This is still maintained but was not thought to be much in use, though as RGM pointed out, it is the 3rd hit in a google search of same. MJI asked for a volunteer to analyse the very red star (well not that red) observed on 27th/28th December with the NB118 filter at various different detector locations for further red leak investigation. So action STH .... <<<< There are possibly more very red star tests using VVV fields around the end of January - watch this space. The imminent advent of v1.3 sparked a request for a bunch of LTO4 tapes of updated VVV catalogues. EGS will make it so and negotiate a suitable transport mechanism. <<<< Apparently the assorted upgrades to the blanks ESO OB status saga were ...... well ...... an improvement, though there are still some genuine blanks from the old days. The good news is that we think the grades written to the files are now (for a change) the same as those received by the PIs. MJI asked about the earlier request for flagging PI data locations on the vistasp/ summary pages, and there are ...... black dots. The later stage polishing of the v1.3 upgrade is proceeding well. In the process assorted handfuls of tile catalogues required Manuel intervention, covering cases where: the original v1.0 had gone walkabout and needed regenerating; catalogues where the original was not quite the v1.0 original; and a few cases where pawprints and tile images also needed some TLC to get them to regrout at all. These have now all been reclassified, rephotometered and are in the process of being cp'd to the respective processed directories together with subsequent assorted header updates. As a further sanity check after doing that we will see from their last update timestamp if any originals accidently got left out. This process will update all tile catalogues before 2012. Quite how far into 2012 products this process needs to go still requires nailing down. <<<< A list of things that have changed in version 1.3 are being accumulate in http://casu.ast.cam.ac.uk/surveys-projects/vista/data-processing/version-log which all involved need to check. <<<< The executive summary is that v1.0 to v1.2 tile catalogues contained unexpected quantities of horsemeat, whereas we can guarantee that v1.3 is 100% bull****. RGM produced some interesting plots of nstds -v- stdcrms for tiles, which among other things highlighted the changeover from one version of the tile astrometry programme to another. This is worth keeping an eye on but we do not expect it to have any noticeable impact on the astrometric accuracy. Last but not least, ESO finally came back with some queries for VISTA Phase3 deliverables from summer 2012! And we thought VISTA Phase3 was closed (well almost) at least for VMC and VHS, though who knows ? 8. VST update -------------- All the VST January processing is finished, and much of February too. The illumination corrections for January await computation though. Raw ingest is as usual running about 4 days behind observations, so no real change there then. The processed/ directory got rather full over the last week so one of the new RAIDS (apm48_b) has become the new suppository and data from 1st February onwards will y va. As noted earlier, but in less detail, the Chilean ATLAS u-band extras (with a different project ID) in the SGC have been processed and morphed in to the earlier iteration of the nightly processed data. For the record these are: 20121013 runs 200-205; 20121025 runs 130-139; 20121112 runs 155-168 = not exactly a massive haul ! The band-merging software is either iterating to a conclusion or proving there's not much need as the VPHAS consortium are quite happily generating separate "blue" and "red" band-merge files (which have a format remarkably similar to the original original CASU band-merged binaries). The Phase3 wishlist from both ESO and PIs est arrivee and the season opens 15th March for a close 30th April. Alle ist nicht noch klar encore une fois depending on how you interpret "The required data products are tile images with their associated weight maps, and the single-band source lists." CASU do not currently produce tile products per se for VST. For single images there's no point due to the big gaps but for dither stacked data you could in principle tile and get a contiguous map + catalogue. Science-wise you gain very little and indeed to date none of the science for VPHAS or ATLAS is using tile products. The other killer is that without correction you would get a serious mess of an illumination correction to deal with either during tiling or after in correcting the tile catalogue - either of which would require further specialist software to sort out. Nigel Metcalfe (ATLAS) found some mystery ra,dec 0.0,0.0 columns in 20110913 and 20110914 VST stacked catalogues. Everything else about these catalogues was fine, so it remains a mystery, albeit fixed to please topcat users. The now expected late request from Oliver (ATLAS) Twist for yet more fields for an (ok admittedly unanticipated) AAO run came in at the previous w/e and as usual coincided with a missing merge files saga due to equally mysteriously missing data from 20120229 - at least that doesn't come round very often. JRF, one of our ex's, asked about making the illumination correction fixing part of the VST software available for list-driven officinados. Feasible, if based on the pre-computed "fix me" information and could be made available as part of the CASUrtools kit, if there's enough interest. A word with Jim to see if he can make it so is called for, but we hestitate to make the Cardinal sin and see if can fix it. 9. Spectroscopic stuff ----------------------- The DM side of the various spectroscopic projects proceeds apace, with special thanks to NAW for generating impressive piles of documentation for the 4MOST and WEAVE design review submissions Meanwhile JRL and EGS swamped the GAIA-ESO community with enough spectra to call their bluff on atmospheric parameter estimation and detailed abundance analysis and the agreeing phase. The delay in the next internal consortium release (DR2), anticipated February March 2013, will slip a bit but being positive it will give JRL a chance to re-examine the spectral PSFs to improve the cross-talk corrections. Assorted interpolation schemes (well they do like it resampled don't they) were tried and tested and since the difference between linear and the cubic spline representative (Keys) was negligible in the tests -> linear it is -> replacing the rather blocky, albeit honest, drizzle. A few low-level (~250m/s) interesting quirks have arisen in pipeline velocity comparison tests between different gratings e.g. HR15 -v- HR21. The culprit is probably equally low-level model atmosphere differences caused by mismatching asymmetries in the different spectral regions (TBC ...... WTS). The end of January Phase3 submission of Gaia-ESO data products got rescheduled due to .... [the following text was expurgated]. 10. ESO in-kind stuff --------------------- JRL with minimal help from MJI has been bombarding ESO with documents and emails faster than c. Extracting the information on where the goalposts are moving has proven to be even harder than normal, but at least two condundrums have crystallised relating to full error propagation and the Naming of Parts (or was that files?). Writing the software, even with CPL boxing gloves on, has proven the easier of the deliverables and is proceeding at flank speed. The design documentation for the VISTA pipeline upgrades has been delivered together with a tarball of the VISTA enhancements so far and a suitable reprocessing schedule is being worked on. AKY is helping out with overviewing reflexing, not to mention Keplering actors. 11.AOB ------ STH gave a brief update on NGTS plus CHEOPS and Echo (the latter competing with Plato for an M3 slot down-select September) Finally, an oldie but goldie, progress on ALL papers will be an agenda item at the next CASU meeting. <<<< Continuing actions ------------------ STH - impart information regarding the radial extent of VISTA diffraction spikes -> web pages EGS - finish comparing VPHAS band-merged products with ours SC - when you get back expedite the handover of the spare LTO4 library unit plus associated 30 TB disk system MJI - plonk down randomly placed apertures to estimate VISTA tile noise STH - update the ZP offsets for the VISTA calibration paper for manana JRL - report on further progess with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights JRL - finish notes of limitations/mods needed for the ESO Giraffe pipeline EGS - finish off the relocation and HDU updates for v1.3 VISTA tile catalogues New actions ----------- MJI - check how far into 2012 the v1.3 upgrade needs to be done ALL - clear up home disk space usage STH - remove the old home disk accounts related to transient stuff SC - investigate possible NFS foilbes causing push-pull transfer asymmetries MJI - check if Carlos is going to the VVV meeting AKY - apply some TLC to the VISTA raw data tape backups MJI - que pasa con el v5 links STH - are the very red star tests for NB118 useful? EGS - sort out the VVV LTO4 tape transfer ALL - check the v1.3 version log text MJI - ensure progress on ALL papers is an agenda item at the next CASU meeting