CASU meeting Tuesday 26th March 11:30-13:00 CASU meeting area. Present: MJI, JRL, AKY, SC, STH, RGM, NAW Apologies: MR, EGS, JPE, JRF Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- JRL - still continues to investigate "rolling" master linearity corrections and will include "average" channels too - eventchewally ! <<<< SC - still needs the high-five Blade Ubuntu/Debian upgrade completion <<<< STH - progress with VISTA calibration papers ongoing and may circulate draft versions in time for next meeting- honest ! <<<< JRL - still needs to check if the info in darks.dqc could have detected the problem VISTA darks on 20110913 <<<< JPE - may have investigated if location of VISTA diffraction spikes with JRF respect to bright stars are readily predictable, butt ...... <<<< SC - encouraged the LTO4 suppliers by pointing out that 1723 single LTO4 tapes were divisible into boxes of 20 aka mod(1723,20)=3 MJI - attended and duly reported back on the ESO VST meeting RGM - the VMC desires have been wished upon us and RGM will define the new files for submission as soon as he gets some confirmation from ESO regarding directory structure - ongoing <<<< MJI - contacted the VMC PI about ESO PR possibilities of the 47Tuc colour pics and then pursuaded Richard Hook to use them, and supplied FITS tiles and sample colour pics JPE - investigation of progress of use of EVALSO was superceded by events MJI - investigated use of DR8 SDSS catalogues in SA fields JRL - ne back-propagated pas improved U-band cataloguing for ATLAS since a cunning more reliable alternative s'apercu JRL - has altered pipeline limits for automatic detector-level stacking EGS - consequently removed surplus-to-requirements stacked products 2. WFAU minutes ---------------- MJI noted that the WFAU grant renewal submission will be in June and is curious to see how the 10K hardware limit is dodged. 3. Hardware update ------------------- Much ordering and arriving of kit including: the 1TB disks for the apm18 upgrade and a couple of spare 3 TB disks arrived; apm47 and apm48 sont arrivees; apm49 has been ordered; and loads of Gaia goodies turned up including an Overland LT05 tape unit + attached CPU and a 108 node disk farm with a mere PB of storage space housed in the Dog Kennel. Meanwhile, back in reality mode, other CASU projects will inherit the cast-off Thumper and spare LTO4 tape library unit. We are still awaiting a new G-bit switch. SC will chase that one up <<<< A couple of vulnerabilities have been spotted with apm25 and apm28 which need addressing, <<<< Rather too many nfs: server apmxy not responding, still trying nfs: server apmxy OK - have been occurring -> que passe ? <<<< MJI asked the assembled Mottley crew to be more cognisant of disk space when reprocessing, since 0 space = lots of unsuspected Murphy-like problems. The erstwhile pre-Cambrian b/w printer otherwise known as apmhp4 has been emitting what can only be described as plaintive mating noises. It either needs replacing or a suitable mate locating. <<<< 4. Meetings and telecons ------------------------- The VMC skypecon, the day after the last CASU meeting, duly occured with much ado about the annexed 47Tuc, among other things. JPE tried his best to preach the gospel of truth about VISTA calibration and related PSF issues but we suspect some of it fell on stony ground. MJI somewhat reluctantly attended a VST-fest at ESO on 21st March at which representatives from ESO Garching, Paranal, Astrowise, VSTube, CASU, KIDS, ATLAS and VPHAS chewed the cud about VST issues and concluded a 4-phase ruminant-like set of stomachs was clearly required too properly digest VST data. Minor foibles aside, the main issue is coming up with a foolproof robust way to cope with the debilitating 0.25mag scattered light/illumination correction problem. This is made up of 4 components, one for each stomach: a radial concentration of scattered light caused by the multiple optical surfaces - this more or less matches the natural vignetting present in the system; scattered light entering from non-astronomcial directions - it is a very open dome structure with lots of reflective surfaces not helped by the limited amount of baffling; the "proud" masking strips covering the r/o electonics all around the outer 4 long CCD edges and the middle (large gap) rows - not only vignetting but also scattering as if to make up for it; and last but not least, random gain variations in several of the detectors. Measuring this effect is also compounded by 10-20% of the frames having rather interesting PSF variations over the field, e.g. relying solely on SA fields to measure the illumination correction may not be such a good idea. CASU's approach is the usual one: monthly twilight flats = monthly illumination corrections = use everything available to monitor and measure i.e. 2MASS, APASS and SDSS. JRL is chasing up acquiring the latest version of APASS <<<< JRF and MJI will be off to NAM to sample the culinary delights of Manchester. JPE and NAW headed off to Chile for the Third VVV Science Meeting, in Vina del Mar and will update us next CASU meeting. EGS is holding the Euclid Fort at an OU-VIS kickoff meeting 26th-27th March a Paris, having come off the back of a 4MOST meeting in Potsdam last week. MJI and NAW will attend the upcoming WEAVE engineering meeting in Amsterdam 17th-18th April. There's also a UKIDSS Science Workshop on the horizon, #IV 31st May-1st June in Edinburgh (http://www.roe.ac.uk/ifa/Meetings/UKIDSS12/index.html), which coincides with the UKIRT Board meeting in Edinburgh. 5. Data archives update ------------------------ Everything is up to date, and most of it even backed up 6. WFCAM update ---------------- Ingest, checking and processing all on track despite the occasional record-breaking large number of files per night. Data to end of February have been checked and are now flagged as ready for copying. The usual culprits required instant access to the processed data. Simon Dye is to be the UHS Project Scientist, the survey (J-band) begins in earnest in June. The raw data for this will not be going to ESO but presumably any remaining UKIDSS survey data (e.g. UDS in the upcoming Autumn seasons) plus calibration data will. MJI needs to double-check this. <<<< To paraphrase the Board's view: 2012/13 will be the first year of the UHS and they have notionally set aside 227 nights for this. The remainder of the time (137 nights) will be for everything else: engineering, UH, CEOU, KASI, DDT, and also some time for odds 'n sods of bits of UKIDSS. KASI, by the way, are a new UKIRT customer, the Korean Astronomy and Space Science Institute, and will be providing a bespoke narrowband filter for their project to study [FeII] outflows in the Galactic Place, as a complement to the UWISH2 project. As there has been no massive stampede for processing the remaining WFCAM missing camera nights any action on this front has been left in abeyance. At the next pause in WFCAM processing, and in anticipation of the final UKIDSS release, the av conf column upgrade will be back-propagated to the earlier processed catalogues. The next UKIRT Board report is due 26th April and the next Board meeting <<<< will be in Edinburgh on 31st May overlapping the UKIDSS science meeting. 7. VISTA update ---------------- Receipt of data, ingest, checking and processing has generally been going smoothly though the latest batch of received disk data produced a lot of corrupt files. Ironically, since 29th February we can now receive both VISTA and VST data using internet transfers from Garching. These currently run weekly but we will be switching to daily soon for both, and we will then run both VISTA and VST processing a la WFCAM style. The VISTA internet transfers require a special CASU account for VISTA data since we need access to all data taken. This has been setup and even seems to work. The internet transfers were used to patch the corrupt disk files, and also as a check, and from now on VISTA data will make the Paranal -> Garching journey via transfers through the EVALSO link. Interestingly VST and VISTA are currently producing similar data transfer volumes (note the x2 difference in Rice compression for 16-bit -v- 32-bit). Raw data up to the end of February has arrived and data up to the end of January have been processed through Phase I of the pipeline. The January VISTA data is being postprocessed but zeropoints and grouting are still to do. Not much further progress in the apcor2 fix, though all the individual pawprints have been corrected and a bunch of missing original tile catalogues created, the Big (Re)Grouting Episode remains to be done. <<<< Not much progress either with the next episode of VISTA Phase III deliverables. The sounds of silence from ESO on this are deafening so in lieu of new, VMC and VHS will be delivering more of same OB-level products shortly, i.e. ~mid-April. EGS generated some interesting VISTA airmass distributions for the public surveys for use in 4MOST simulations. Mike Read has made some impressive VVV+GPS mosaics which will be used as part of a VDFS press release for NAM. 8. VST update -------------- As noted earlier, since February 29th Internet transfers of VST data from Paranal -> Garching have been using the EVALSO link and VST data is now available ~24 hours after the observations. Currently VST data transfers from ESO to CASU make use of this but are batched in weekly chunks. However, given the usual keen interest in rapid turnaround this will become daily shortly and the current backlog of processing will be gradually reduced. <<<< Meanwhile the VST web pages, processing and survey progress have been upgraded to reflect the full status of what is happening. The VISTA ones need a similar makeover. <<<< The ATLAS survey is currently hammering through NGC strips and as usual a 2df observing run scheduled for the end of April urgently requires the processed data for target selection. After an internal review of the current VST processing, a few improvements, mainly to the master bad column flagging, have been implemented which required restacking and recataloguing of all the data to end of January. February onwards already has the improvements incorporated. When this is finished a fix for the scattered light/illumination will be implemented using a mixture of 2MASS, APASS and SDSS calibrators. As the fix is so complicated this will involve producing a "fixed" version of all the relevant catalogues whereby all the internal flux values are corrected. After reclassifying and recalibrating these will form the basis for the standard user end products. The original catalogues will also be available, mainly to simplify implementing future improvements to the scattered light correction. After much trial and error we have concluded that ab-initio detection on U-band data is not good enough for the ATLAS requirements and have decided to complement the normal U-band catalogues with list-driven catalogues based on the g-band. These will be generated at the band-merging stage and will be included as an extra in addition to the original u-band catalogue. Whether or not to extend this, or some variant of the list-driven approach, to the other bands remains TBD. The VST OB grades will shortly be available and will be accessed from ESO and then incorporated into the FITS headers. 9. Spectroscopic stuff ---------------------- There is a lot of action on the spectroscopic front involving Gaia-ESO, a 300 night programme which started in January on the VLT, and is generating a lot of MOS GIRAFFE and UVES data. Complementary to this are development projects for possible future large scale MOS instruments on 4m telescopes, 4MOST in the south and WEAVE in the north. As MOS processing is becoming a recognised part of the CASU programme this will be regular agenda item in future meetings. JRL has already established a prototype processing pipeline for GIRAFFE data which will form the basis for any future MOS processing requirements. This is being actively used and developed for the Gaia-ESO project. In addition to operating this pipeline, CASU are running the ESO GIRAFFE pipeline and assessing its foibles (for later feedback to ESO) and are also cross-comparing pipeline results with a dedicated pipeline for GIRAFFE data developed at Keele. As usual, part of the requirements for this are to generate some coherent documentation describing data formats etc... and also as usual WFAU are themselves working on a draft ICD and GES schema. Hopefully we'll all converge. EGS has been analysing simulated spectra for 4MOST using a generic s:n measuring programme dug out by MJI. Most of the science either revolves around EW estimates and RV measures, and using theoretical error bounds and measured s:n gives a quick assessment of spectra quality. 10. ESO in-kind stuff --------------------- Not a huge amount of action on this front yet. One of apm47/48 will be the server for this project and will begin by hosting the accumulated HAWKI data, which has gradually been transported here by USB bricks over the last few months. The stuttering speed of transfers appears to be due to the fact ESO have to uncompress every image first, then update the header and then recompress it. The native raw HAWKI data is a strange 32-bit real bodge which unfortunately means that lossless Rice compression is not viable. JRL found a cunning alternative supported in CFITSIO that gzips the image components but keeps all the FITS headers native. Although it is not as fast as Rice compression, the HAWKI data here is stored using this compression method Next in line for transfer and ingest is the VIMOS data. The VISTA pipeline upgrades component of the deal awaits delivery to us of CPL version 6.0, which just to stand still will initially require a host of small changes to the already delivered VISTA software. 11. AOB/news ------------ Favourable reviewer's comments for the CASU grant submitted end of November were recently received. Detailed answers to the points raised were concocted and sent back and a grant offer was made shortly thereafter, aka the day before this meeting. MJI processed a weeks worth of INT WFC data for our old friend Bob Abrahams in return for a challenging in-kind delivery. The rubbish opening/closing mechanisms for the upper south-facing windows in most of the APM offices have given up completely, not helped by the repair guys leaving the kit half disassembled and then scarpering. An alternative more primordial solution is on the cards. The recent tree clearing on-site near the APM building revealed a yellow orb and the lack of blinds on the aforesaid upper windows. Cunning blinds have been ordered. Continuing actions ------------------ JRL - investigate benefits of a "rolling" master linearity correction or even an average channel one SC - finish off the rest of the Ubuntu/Debian upgrade for Blades STH - progress evermore the VISTA calibration papers writeups and circulate draft papers before the next CASU meeting JRL - check if the info in darks.dqc could have detected the problem darks JPE - investigate if location of VISTA diffraction spikes with respect to JRF bright stars are readily predictable RGM - supply list of extra VHS deliverables New actions ----------- SC - chase up acquiring a new G-bit switch EGS - scope the fixes required for apm25, apm28 and get help as needed SC - investigate if any faux pas are causing the multitude of nfs whinges SC - order a replacement B/W printer for apmhp4 JRL - keep chasing up the latest APASS catalogue MJI - check if ESO will get access to WFCAM calib+UKIDSS data after 1st June MJI - write and submit the CASU report for the next UKIRT Board meeting EGS - finish off fixing the apcor2 problem for VISTA grouted tile data AKY - reduce the VST processing backlog as much as possible JRF - get the VISTA processing web pages to reflect the full processing status