CASU meeting Wednesday 29th August 11:30-13:00 APM Present: MJI, JRL, AKY, RGM Apologies: EGS, NAW, STH, MR, JPE, SC Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- STH - all quiet on the VISTA calibration paper front i.e. still awaiting the circulation of a draft and the Y- and Z-band Vega zero-point offsets for persual <<<< JPE - characterising the "radial" extent of diffraction spikes for VISTA JRF pawprints also being worked on by STH + PhD student from the pov of STH variability aspects - still <<<< SC - the new G-bit switch was installed and web page monitoring updated MJI - was handed a copy of UCAC4 post-meeting but still unclear if PPMXL fits the bill as an alternative - TBI <<<< MJI - the "averaged" linearity curve test VISTA night gave similar results to the original which is a good job as from 1st August we have switched to using "averaged" MJI - updated bandmerging prog for VPHAS data ongoing but not rapidly <<<< MJI - investigated astrometric distortion patterns seen in tiles which solved some puzzling results (see later) JRL - the VST video board change only caused a 1-2% change in internal gains so not much point reprocessing the last 2/3 of June VST data 2. WFAU minutes --------------- Creative use of ordering pawprints with different exposure times in deep stacks prior to creating tiles lead to a few problems and email exchanges. The problem happened because although the stacking code is designed to cope with arbitrary exposure times it scales everything to the reference file (i.e. the first one in the stack list). The "random" order of component files in the 6 deep pawprint stacks that made up a tile lead to different effective exposures times in these stacked pawprints. Creating this unbalanced pawprint set explored internal scaling options in the mosaicing code that were not considered as possibilities for VISTA tiles and uncovered a bug in the CASUtools mosaicing software. The bug only featured if different effective exposures times are used in the 6 pawprints that make up a tile. As this doesn't occur in normal pipeline processing it hadn't been spotted. JRL has fixed the bug and updated the software. The other issue that initiated a few more emails was apparently missing HDU items, in this case "SEEING" - although actually a whole block of headers. This was laid to rest by directly comparing the original file headers, where all were present, with ones in use at WFAU, where they weren't. The extra history records and the "CU8" were a bit of a giveaway. Not sure if this was in the minutes or we got it from our spy recently arrived in Durham, but we are pleased to hear that WFAU will be stacking the U-band Chilean with Tom's ATLAS equivalent. 3. Hardware update ------------------- The sundry items of surplus computer kit were finally moved to the Dog Kennels even including STH's infinite machine. With the installation of the many Dalmations + friends, MJI tasked SC with expediting the handover of the spare LTO4 library unit plus associated 30 TB disk system to take us to the apm half-century and a celebration. <<<< There was a bit of discussion on the progress of using postgres on apm49 to replace the sybase server apm5. EGS will provide an update on progress at the next meeting. <<<< After the new G-bit switch was installed a couple of old Netgear switches were removed and their sundry network connections transferred across. So far so good, aka nobody has complained. SC provided the following summary of the main network connectivity: 10 Gigabit nete to netb Hoyle = the original link from the Hoyle to the central server room (nete) 10 Gigabit netn A7 to netb Hoyle = the new link to APM room 7 (netn) from the Hoyle 10 Gigabit netn to net29 = the new link from netn to net29 (the link from nete to net29 has been removed) On netn, ports 17-32 are on VLAN290, so are for 172.31.x.x addresses (the lights-out management of the chassis and blades, and the gaia internal network). As it stands, traffic from room 7 to the central server room goes via the Hoyle (10Gb), but most of the heavy traffic should be within room 7 anyway. The monitoring of the new links is done via the netn ports. 4. Meetings and telecons ------------------------- Nobody present had taken part in the VMC skypecon on 26th July so we had to resort to reading the minutes from it. The most relevant bit relates to the astrometric distortion (mainly affecting tiles) that RGM reported on at the last meeting. This aspect is discussed in the VISTA section in these minutes but its worth issuing our usual salutory reminder that for accurate proper motion work it is better to work with the pawprints and not the tiles. The next VMC skypecon is 20th September. There was a long and fruitful 4MOST telecon on the 8th August to discuss the progress of the simulations and to cap the amount of effort required to generate the results. Upcoming meetings include: EGAPS in Warwick 10-11th September; a VHS one-day meeting at the IoA on 12th September; and the ESO surveys meeting October 15-18th in Garching. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ All up to date, much UVEX and Kepler band-merging done and the latter served up. This was followed by generation of 2012 IPHAS data, and then a complete remerge of all IPHAS to use the surface fitting option and the updated photometric calibration for 2003-2004 data. 6. WFCAM update ---------------- Raw data transfers and ingest all up to date. Data processing and releases completed for June and July, whilst August processing is on track. The ~10mas systematic difference in astrometry noted previously between CASUtools software and the WSA listing was probably caused by not pushing the ZPN inversion hard enough in the CASUtools subroutine that does this. This has been upgraded to use double precision in the inversion and an extra iteration and now produces sub-mas precision in forward<->backward transformation tests. These previously came in at a few to several mas precision. The upgraded sky subtraction strategy to work around some of the pawsky problems with LAS-type patching observations has been incorporated into the pipeline for August processing as a more extended test. So far so good, but even better is to observe complete tiles for shallow surveys. With UHS data taking in full swing, MJI asked EGS (in absentia) to add a UHS progress web page to the current UKIDSS suite. <<<< Watson Varricat came up with an interesting problem relating to WFCAM and crowded field photometry, e.g. as in the Galactic Plane. Comparing repeat observations in similar seeing yielded the expected magnitude difference as a function of magnitude. However, repeating this exercise with observations taken in significantly different seeing yields an apparently non-linear magnitude difference -v- magnitude. This is a generic problem for all crowded-field photometry and is due to the estimated local background level being seeing-dependent. The estimated level is always incorrect, its just more incorrect when the seeing is worse. This effect is inherent in virtually all methods, even PSF photometry. It is possible to estimate the size of the effect by comparing photometry from different apertures as a function of magnitude and in principle, at least, go some way toward correcting for it. There were the usual suspects requesting early data deliveries and one old friend turned up - a catalogue w20120505_01819_sf_st_cat.fits which had several NaNs in extension #3. Quite why the NaNs disappear when regenerating the catalogue remains a mystery, but they do. 7. VISTA update ---------------- Aaprt from the odd data transfer hiccup, ingest and tape backups of raw data are proceeding apace. It may not look it on the web pages but the main processing is up to date and proceeding around 1-2 weeks behind the observations thanks to JRL taking over for a bit after JRF's departure. After a refreshing break, AKY will take over VISTA processing from JRL (MJI is currently doing the honours for VST processing). Releases await the return of various folk from holiday but June should be ready to go shortly and July not far behind, <<<< though MJI needs to compute the July photometry updates first. <<<< Whilst puzzling over the rapid fill up of the latest processed suppository disk an interesting quirk of the VISTA squishing script was uncovered which goes some way to explaining the mysterious vanishing images that had plagued us since February. Squish once fine; twice correctly checked if already squished but malheuresement then removed image - d'oh! The rapid fill up was due to whole pile of leftover proceessing debris, duplication of 20120601 in 2010531, and a whole bunch of bits of nights with unsquished data present. Anyway, upshot mystery solved, and lots of disk space recovered. JRL emailed Thomas S. about the paucity of twilight flats being observed which makes generating monthly masters more challenging than it should be. Terrapix are now downloading vast quantities of UltraVISTA data and clogging up the VISTASP server for others, another example of cause [greedy b*******s] and effect [impatient b******s]. Bo Milvang-Jensen demonstrated that people do actually use the CASU VISTA QC tables by asking for them to be updated. EGS will oblige. <<<< Maria-Rosa forwarded what can best be described as a questionnaire for information describing the properties of the ESO Phase3 delivereds. This was briefly discussed and includes several quite technical questions. For now mr., and no doubt other PIs, have just pointed to various bits of our technical web pages to cover this. However, we suspect this will be bounced back before the next Ice Age. Meanwhile Phase3 deliveries progress: the VVV upload is finally finished; VHS is not far behind and news about VMC and VIDEO awaits the return of EGS. [EGS confirms that required VHS, VMC and VIDEO data have all been delivered.] MJI spent a while puzzling over the astrometric distortion patterns reported by RGM, mainly in tiles but also in systematic differences between embedded RA, Dec columns and the same computed from X-Y + WCS information. To cut a long story short, apart from small effects due to focal plane adjustments and fine tuning of the PV radial distortion terms in Autumn 2010, the main effect is due to a cockup in interpreting the definition of ZPN as set out by Calabretta and Greison (2002). The good news is that this "feature" does not affect pawprints for WFCAM, VISTA or VST, nor does it affect any project using solely CASU software for its astrometry or solely WCSLIB for its astrometry. Unfortunately, in creating VISTA tile mosaic images we mix'd and match'd one small bit of code to improve the speed. This directly imprints a ~+/-100mas distortion pattern in the image tracking the 6 pawprint footprints in a VISTA tile. In a nutshell CASU defined ZPN as CASU r' = tan(r) r" = k1*r' + k3*r'**3 + k5*r'**5 i.e. project to standard coordinates r', then distort where r is sky angle in radians from optical axis and r" would be distorted standard coordinate radial angle on detector focal plane. We thought Mark Calabretta had done the same in WCSLIB but no. Instead Mark r' = k1*r + k3*r**3 + k5*r**5 i.e. distort angle first, then we thought he converted to standard coordinates but NO again, these are his equivalent, i.e. the projection bit is all in the ZPN part and indeed empirical tests suggest his projection is as it says in the "paper" tin and thereby different, to a good approximation, by tan(r)/r If you are used to working in standard coordinates, distorted or otherwise, this makes for some interesting subtleties in converting back and forth between celestial and detector coordinates. We've always suspected the occasional oddity in VVV photometric calibration, particularly in Y and Z. Roberto Saito has been examining the uniformity of the global calibration in VVV data and compiled a shortlist of fields (tiles) with obvious problems. Most of these are simply due to poor weather during the sequence of observations but a few are caused by poor calibration in high extinction regions, particularly those with patchy 2MASS coverage. A simple extra trap for the decision boundary of which of the 3 methods to use fixes this and suggests it would be worth a version update of all VISTA photometric calibration to v1.3 to fix this and also bring all the calibration to the same version. However, before doing this we are awaiting the final word on the colour equations and extinction correction being examined in the VISTA photometric calibration papers. 8. VST update -------------- Raw data transfers, as for VISTA suffer the odd hiccup but generally are fine. Ingest, processing and release (July) are up to date, though it transpires we are tad behind with raw data tape backups. This will be addressed when the spare LTO4 unit is handed over. <<<< The one VST foible not yet addressed, are the random gain changes in some of the detectors. Independent ab-initio fixing of these robustly is not possible but may be feasible at the band-merging stage, particularly for VPHAS, though ATLAS will be more challenging. This is another motive for progressing the more general purpose band-merging software update. The ongoing request from Richard Hook for more PR images from VST, in particular from VPHAS, was neatly sidestepped by handing over this subset of PR magic to Nick Wright who will shortly be arriving at U. Herts. Danny Steeghs (VPHAS) found an interesting VST processing cockup of 4 nights of April U-band data. The symptom was weird U-band zpts, the cause was confidence maps had somehow got copied over (rebadged presumably otherwise the pipeline would have objected) and used instead of flats .... hmmmm..... The VST video board replacement covered detectors 81 to 84 and all we know is that tests started on June 4th and the ticket was closed on June 7th, with some swapping of cables etc... However, comparing flats taken before and after doesn't really show up any serious changes so for now we will run with the data as processed at the time. fitsio_ccdproc g_SDSS_flat_20120603.fit flat=g_SDSS_flat_20120705.fit Gain relative to #1 1.000 0.953 1.047 1.020 1.068 1.054 1.076 1.106 0.967 1.046 1.019 1.102 0.973 1.009 0.965 1.017 -------------------------- 1.029 1.163 0.995 1.022 1.065 1.033 1.008 1.007 1.177 1.213 1.117 1.072 1.056 0.996 0.946 1.036 fitsio_ccdproc g_SDSS_flat_20120705.fit flat=g_SDSS_flat_20120603.fit Gain relative to #1 1.000 0.948 1.041 1.012 1.062 1.046 1.060 1.087 0.960 1.029 1.002 1.078 0.969 1.000 0.956 1.003 -------------------------- 1.022 1.159 0.993 1.024 1.051 1.020 1.001 1.006 1.163 1.203 1.101 1.066 1.051 0.996 0.946 1.041 Updating the photometric calibration awaits the next (final?) installment of APASS at which point we will also add in an AB mag calibration based directly on APASS as an alternative to the Vega-based one. If the next APASS version is delayed though we might try this with APASS v6 since the zpt calibration is now the weak link for VST photometry. 9. Spectroscopic stuff ----------------------- GaiaESO - pipeline velocity comparisons look good, various fixes to the stacking have been made, plus a few more tweaks to the release DB progess. 4MOST - round 4 simulations have progressed well and even round 5 is now out, though it needs looking at. WEAVE - the feedback from the interim review was positive and not many actions, even better. 10. ESO in-kind stuff --------------------- The quarterly report and design doc for the VISTA pipeline upgrades were delivered to ESO well over a month ago - the silence is deafening. HAWKIng - AKY expressed an interest in learning about and helping out with the reflex workflow and its keplerian actors - no DM there then ! 11. AOB/news ------------ The CASU job finally got advertised with a closing date of the end of September. The IoA BBQ is almost upon us. Upper windows battle with security proceeds apace .... and we thought we'd won! CASUtools: it appears that imcore doesn't like it if you don't give the output file with an extension of either '.fit' or '.fits' and seg faults. Well what do you expect .... Jim'll fix it. assorted upgrades to mosaicer (exptime and WCS use = noticeable) and stacker (WCS = extremely subtle and unnoticeable) to fix CASU foibles Continuing actions ------------------ STH - progress evermore the VISTA calibration papers writeups, plus VISTA Y and Z Vega offsets and circulate draft papers before next CASU meeting JPE - continue to characterise radial extent of diffraction spikes for a STH range of values of NDIT*DIT for pawprints, and thence tiles MJI - update bandmerging prog to cope with weird VPHAS observing pattern New Actions ----------- MJI - read on UCAC4 DVD and see if its worth it cf. PPMXL SC - expedite the handover of the spare LTO4 library unit plus associated 30 TB disk system EGS - provide a progress update of sybase apm5 -> postgres apm49 EGS - add a UHS progress web page to the current UKIDSS suite EGS - release June VISTA data MJI - compute the July photometry updates EGS - update the CASU VISTA QC tables MJI - sort out backlog of raw VST LTO4 backups