CASU meeting Wednesday 26th September 11:30-13:00 APM Present: MJI, JRL, AKY, STH, EGS, RGM, NAW, JPE Apologies: MR, SC Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 1. Actions from last meeting ----------------------------- STH - produced a draft of the VISTA calibration paper (see later) JPE - have characterised the "radial" extent of diffraction spikes for JRF VISTA and it was suggested that this information goes in the STH technical web pages for users to make their own use thereof <<<< MJI - cf. UCAC4 and PPMXL as alternative astrometric catalogues ongoing <<<< MJI - updated bandmerging prog for VPHAS data ongoing and not rapidly <<<< SC - is still expediting the handover of the spare LTO4 library unit plus associated 30 TB disk system <<<< EGS - provided a progress update of sybase apm5 -> postgres apm49 et al. EGS - adding UHS progress web page to the current UKIDSS suite ongoing <<<< EGS - released June VISTA data MJI - computed the July photometry updates EGS - updated the CASU VISTA QC tables JRL - sorted out the backlog of raw VST LTO4 backups 2. WFAU minutes --------------- All quiet on the Northern front apart from VST broken catalogues. Most of the latter were caused by the unlikely switch of U-band confidence maps for flats for the first 4 nights of April processing noted in the previous CASU minutes. To prove we read the minutes, we took note and a sprinkling of bon mots francais have been added herein. 3. Hardware update ------------------- The management of the apm28 and apm37 Pathgrid machines was briefly discussed together with a wish/unwish list of required system software. The donation of the LTO4 tape libray unit and the 30TB disk is becoming more urgent due to lack of disk space. The postgres tests on apm49 were successful which lead to a further discussion of which machines do what on the cluster aka plone and assorted DB serving for VISTA, VST, WFCAM and Gaia-ESO. The recent upgrades to the mosaicing and stacking software were released via CASUtools. 4. Meetings and telecons ------------------------- MJI attended the EGAPS meeting in Warwick 10-11th September to find out how the latest INT WFC IPHAS, UVEX and Kepler releases, together with the VST VPHAS data releases, were being used and if any new problems had been uncovered - none thankfully. The efficacy of the VST Video board replacement in early June, to get rid of the detector#82 problem, was checked out during the meeting. Prior to the June intervention around 25% of the VPHAS observations showed random gain changes for this detector, varying from roughly -0.2 mag to +0.5 mag. These jumps have occasionally been seen even during readout. In the two months of data examined after the intervention no gain changes were seen. The VHS meeting at the IoA on 12th September was well attended and some satisfying science results shown. A few well known issues such as rare objects searches being sensitive to spurions were aired, but the more significant concerns were: astrometry systematics on tiles, now a known issue and which was covered in the previous CASU minutes; slightly worse seeing in tile products compared with pawprints. This is almost undoubtedly caused by a combination of WCS solution errors and the extra gross interpolation required to form a tile. As the default 2 arcsec diameter aperture corrections change by less than a 1% between tiles and pawprints it is presumably just the core of the PSF that causes this seeing change. The slight degradation in the core of the PSF washes out by a radius of 1 arcsec and hence has negligible impact on the quality of the photometry; lower than expected signal:to:noise improvements in tiles -v- pawprints; MJI suspects that this is caused by the difficulty of accurately measuring the underlying rms noise due to varying noise covariance in heavily interpolated data (tiles particularly). He will investigate by plonking down randomly placed apertures to compute the background noise directly to see if we get the expected improvement of ~0.4 mag rather than the inferred ~0.2mag from the pipeline noise estimates <<<< There was the usual monthly VMC skypecon on 20th September featuring no new surprises and the usual skypecon inefficiencies. Upcoming meetings include: Computing Open Meeting 2nd October which needs some CASU representation <<<< the next VISTA IOT on 22nd October ESO surveys meeting October 15-18th, EGS, MJI, JPE and RGM will attend ADASS in Urbana 4th-8th November 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ The EGAPS meeting highlighted the need for UVEX propagating to the band-merged products and the whole lot serving through web pages. <<<< A couple more IPHAS runs were found, processed and shifted out of the door - is that it for now ? All the IPHAS data was dumped to a 3TB USB disk and handed over at the w/e. 6. WFCAM update ---------------- Raw data transfers and ingest all up to date. Data processing for August is finished whilst September processing is on track. There were the usual early data release requests that were dealt with as they came in. The final release of August data got delayed by two "interesting" <<<< technical issues that arose over the last month or so and took a fair bit of effort to investigate. The first issue related to strange MSB sequences spotted by MJI in UHS tiles. These were characterized by incomplete tile coverage (i.e. not all 4 pawprint pointings were present) in the middle of sequences of otherwise normal tile observations. After a few manual tests and some surprisingly complex scripting, it turned out that around 40hours worth of MSBs were affected and need repeating. The problem appeared to be caused by a recent change to the translator (the software that converts MSBs to sequences that the telescope and instrument can read) and not the SDT or the MSBs themselves. This problem was fixed by JAC on 7th September. The second issue relates to subtleties introduced by using the pawsky mask option when generating sky frames. For surveys that are designed with a large number of jitters per pawprint with reasonable spatial coverage this is much less of a problem. However, when used as a backup option in lieu of incomplete tile coverage in say patched LAS observations, the effects can be significant. This issue came to light during checks for DR10 by SJW. He noticed occasional "shadowing" around large galaxies which is one of the consequences of only using 1 pointing and a small number of dithers to compute a sky estimate. The pipeline sky estimation strategy hasn't changed but the sequence of observations had and this triggered the switch to pawsky masking rather than tilesky correction. The effects of the pawsky method on large and/or bright objects are fairly obvious when examining the images, or indeed skies used. However, the other effects are more subtle. It is well known that no matter how good the masking algorithm, for single pointings sky estimation based on only a small number of frames and dithers leaves some residual object flux, below the pixel-level detection limit, in the sky frame. The residual is diluted by a factor O(1/n) when doing clipped averaging but still leads to oversubtracting faint outer profile wings by this amount and thereby introduces a small systematic bias which progressively increases at faint magnitudes. The good news is that for point sources significant differences only arise for heavily saturated objects. Even for extended sources measured in the default aperture there is very little difference. However, the bad news is that when using Petrosian fluxes, where the extended wings have much more impact, its a lot worse and the bottom line is that a large fraction of ALL extended sources are noticeably affected. The only way to fix this problem is to reprocess LAS frames sky-subtracted in this way. This only (sic!) affects some LAS J-band data taken after May 2011, but as this selective reprocessing involves a large amount of bookkeeping it is on hold for now. MJI will put a note on the web pages together with some example plots. <<<< 7. VISTA update ---------------- The occasional hiccups in raw data transfer have been alleviated by increasing the 2 day delay in initiating transfers to 4 days to improve reliability. The same extra delay also had to be used to make VST transfers more reliable. Missing files from nights were an interactive pain to deal with and so far this extra pause before transferring seems to have solved the problem. Releases are running a tad behind as usual during the Galactic Bulge season due to the huge catalogue volume of Ks-band data. Primary processing for July and August is completed we just await the grouting to complete for July and MJI to extract his digit and generate the photometric zero-point updates for August. <<<< JRL commented again on the paucity of twilight flats whinge and having heard back nada from TS will raise encore une fois at upcoming VISTA IOT. <<<< RGM noted that it would be useful to flag PI data locations on the VISTASP summary pages. Tout le monde agreed and EGS shall make it so. <<<< There have been occasional complaints about having to wait more than 5 mins to get the files from the VISTASP server. These occur when the queue gets jammed either due to gigantic download requests or to gremlins in the pipes. Une grande jam occured because of a problem with one of the requested files which apparently did not have a SKYSUB header keyword. These and other amuses bouches are being sampled to find common cause if any. The VVV folk have more or less given up in trying to transfer files across large bodies of water and some LTO4 tapes for VVV will be hand-carried to the Survey meeting ESO for deliverance. JPE reported on some earlier pseudo-action items relating to issues Viking and VISTA. The former involved Terapix downloads which were halted - Will has apparently not yet discussed with Terapix or agreed they should continue downloading. The other Viking issue related to Phase3 deliverables - VSA did these for the last release and Will is not intending to ask CASU for the current one. The VISTA item related to major interventions where the camera was adjusted/reshimmed which may or may not relate to WCS changes. The one in question is at http://www.eso.org/sci/facilities/paranal/instruments/vircam/news.html and shows it was 2010-04-24/2010-05-06. RGM reported back on victory in the Phase3-unique-source-identifier-in- -VISTA-survey-catalogues question. VVV officianodos have been comparing asses and elbows in all things VVV and came up with a short list of photometric oddities. Most of these turned out to weather-related but it did highlight a few pawprint/tile issues in Z-band GP fields with high patchy extinction. This injects large variations in individual pawprint zpts and hence slightly merde alors tile photometry. These need fixing at some point together with all the other leftovers e.g. aper2 for some tile grouting, which will be saved up for v1.2 -> v1.3 updates. An extra sanity check criterion on the fraction of 2MASS stars being used at the calibration phase seems to trap these few problem Z-band fields and will be incorporated during the update. All still awake then knuckled down to reviewing STH's draft of the VISTA photometric calibration paper. Malheureusement, prior to getting seriously stuck into that RGM and JPE had a fascinating discussion on the merits or otherwise of defining Ks VISTA system as Kv and likewise, one supposes, Kw usw..... that did for everybody else. The paper is progressing well and lead to a bunch of suggestions but none too onerous. STH pointed out a couple of curious features that need further followup, at least one of which may be due to using an earlier photometry calibration. A long-standing mystery, whereby tile zero-points are always 1% (i.e. 0.01 mag) or so larger than pawprint zero-points, lead to a lot of convincing, at the time, but complete red herring, explanations. This was eventually figured out post-meeting and is essentially due to the fact that the tile PSF is slighty more blurred in the centre than pawprints (aka the earlier tile-v-pawprint seeing issue) and the way the tile system is referenced during grouting. The aperture corrections take care of this already (i.e. they too are 1% or so larger) such that the reported magnitudes of objects are correct. The offsets between the VISTA system and Vega are still being fine-tuned, but apart from a mystery offset in Ks these are pretty much as expected. STH will update the paper encore beaucoup de temps for the next meeting. <<<< MJI had what he thought was a quick question relating to access of some short exposure commissioning data taken of NGC253. He will forward the relevant bits of email to JPE. <<<< 8. VST update -------------- As we were running out of time, this ended up as the last item, remaining odds 'n sods being held over to the next CASU meeting. Raw data transfers , ingest, processing and release (August) up to date and tape backups too thanks to JRL. The one VST foible not yet addressed, are the random gain changes in some of the detectors. The EGAPS meeting had nice plots of gain variation of #10 including its effects in the odd flat. This lead to a check at CASU of all the VST flats for gain variations in #10 (and others) using the illumination correction information where there is a detector-level set of offsets listed. So for example: NB_659_flat_20111219.fit has an offset of 0.191 mags applied in the illumination correction for #10, agreeing nicely with that found by the EGAPS team. On the other hand extension#14, which they claimed was off has -0.0153 so its probably just the normal gain difference for that one. The normal variation is < +/-5% so being conservative in searching yields only 3 suspicious examples: -0.05 in g #10 for feb2012 0.06 in r #20 for dec2011 0.19 in NB #10 for dec2011 The one puzzling thing found was no offset for u_SDSS_flat_20120414.fit (0.003 #10 and 0.017 for #20) which had also looked suspicious. This suggests that on "average" for that month it was "off" since indeed the mosaiced flat (made to double check) is different to the previous month. From a cataloguing point-of-view the illumination correction takes care of all of this anyway so there is not much point in reprocessing any of these. From an image mosaicing point-of-view there is a keyword (percorr aka offset in mags) in the extension headers that the CASUtools mosaic code uses to correct the overall level. A possible solution here is simply to populate that keyword with the detector level offsets found in the illumination correction process and see if it cures rather than hinders mosaicing. Continuing actions ------------------ JPE - put information regarding the radial extent of diffraction spikes for a STH range of values of pawprint NDIT*DITs on web pages MJI - compare UCAC4 and PPMXL as alternative astrometric catalogues MJI - update bandmerging prog to cope with weird VPHAS observing pattern SC - expedite the handover of the spare LTO4 library unit plus associated 30 TB disk system EGS - add UHS progress web page to the current UKIDSS suite New Actions ----------- MJI - plonk down randomly placed apertures to estimate VISTA background noise ??? - attend IoA Computing Open Meeting and report back EGS - release UVEX data through usual web pages MJI - finish off August WFCAM zpt update and release data MR MJI - add note to web pages regarding pawsky limitations MJI - do the August VISTA zpt update computations and boot July and August AKY out of the door as soon as possible EGS JRL - whinge at next IOT about paucity of twilight flats EGS - flag PI data locations on the VISTASP pages STH - update the VISTA calibration paper for next CASU meeting MJI - forward emails about access to commissioing data for NGC253 to JPE