CASU meeting Wednesday 23rd October 11:30-13:00 APM Present: MJI, AKY, STH, JRL, GM, EGS, CGF, NAW Apologies: MR, RGM, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- Agenda item: progress on ALL papers - had a meeting about calibration paper STH - further detailed analysis of VVV Bulge and disk fields is ongoing to MJI examine the extinction over the entire region. Better extinction maps CGF have been generated and analysis of the extinction calibrator coefficients and the zero-point offsets is ongoing <<<< JRL - replacing apm5 Sybase server -> no more progress recently so EGS still ongoing <<<< MJI - VST dome flats-v-twilight flats keep watching this space <<<< SC - is still ordering another fibre channel raid for apm49 and going for the record between start and finish <<<< SC - has finished Blade system updates, a few odds and software ends are being completed by JRL SC - investigated yet again usage of home disk space and cleared out MJI some debris to alleviate the immediate problem JRL - found out more about state-of-play of WFCAM negociations at ADASS EGS - propagation of HAWKI QC diagnostics to web pages awaited the MJI return of EGS but now awaiting updated versions with better colour equations <<<< 2. WFAU minutes --------------- No comments 3. Hardware update (and software) ------------------ The UKLight connection between the IoA and UCS did indeed change to using the new fibre and ceased working at about the same time. It was fixed eventually though. 4. Meetings and telecons ------------------------- VMC monthly - the change of time/date to Monday 1-2pm was the latest excuse for not remembering to take part. ADASS was endured by JRL, EGS and NAW. JRL reported on the WFCAM news and on the annual HDF -v- FITS debate. MJI was relieved to learn that FITS will outlast him (probably) NAW and EGS will attend a WEAVE DM meeting at IAC 24th,25th October to finalise discussions on who will be responsible for what DM-wise. JRL noted an upcoming GES meeting in Florence 11th,12th November and will report back any relevant news at the next meeting. <<<< 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ Hywel from IPHAS central asked about completeness tests for imcore, a timely reminder that we did a lot of these tests about 10 years ago pre-wfcam days http://www.ast.cam.ac.uk/ioa/research/vdfs/docs/reports/simul/index.html but haven't done that much since. Part of the reason we made the cataloguing code available was for people to do exactly these sort of tests. Part of the problem with completeness determination is that it is very specific to the dataset and how it is to be used since it depends not only on detection probabilities but also on classification reliability and crowding and how these vary over the survey. However, to first order the classification and detection probability completeness is analytically scalable using a semi-empirical model based on a s:n=10 fulcrum. Likewise crowding effects can also be approximated analytically. Although these are by their nature only semi-quantitative they can often be used to short-circuit long numerical simulations. The archives are all up to date though the ING archive needs a new disk space home. <<<< 6. WFCAM update ---------------- There was a bit of a shortage of new flats taken end of September/early October to switch over in the usual manner so October processing is being done with a hybrid of new/old flats, otherwise all is on track. KASI's -JJ has been pushing the skyflip recipe somewhat harder than it was designed for by putting mutiple repeats in the same MSB. The poor old pipeline got a bit confused about how to deal with these correctly and, to cut a long story short, failed miserably. This affects several hundred files on nights of 20130802, 2013090[4-8] and 2013091[3-4]. It looks like the pipeline was totally confused by the 3x repeat of the observations within an MSB and effectively deprecated most of them before proceeding. In this mode it was only designed, or expecting to find, one set of object/sky flips within an MSB, essentially assuming further stacking would be done outside of the pipeline. A simple workaround is to artificially split the MSBs into subunits of one set of object/sky flips by changing the GRPNUM keyword value and indeed that works as intended. Reprocessing of these chunks of data is proceeding apace. There was a minor whinge from David Stark about dark patches in UKIDSS images. The larger ones are at the +/-few ADU level which given all the corrections applied is about as good as it often gets and smaller ones are usually reflection/emission from "dust" not present on say the monthly flats that only get partially removed during sky subtraction depending on background illumination conditions at the time. We have informed ESO that deliveries of raw WFCAM data will cease after 31st December 2013. If that is not the case shout loudly. 7. VISTA update ---------------- The receipt of raw data proceeds apace but processing, particularly the post-processing tasks, have been stalled slightly awaiting the Blade software/firmware updated. As an interim measure some scratch space on apm45 and apm46 was made available for post-processing. It was noted that we need to keep up to date the list of fixed updated tiles and pawprint stacks found whilst checking VVV data which should all have a v1.3.1 in the header. <<<< The saga of the VIKING reprocessing by Terapix appeared to be purely motivated by photo-z seeing-matched requirements and better recovery of the large-scale lower surface brightness outer regions of galaxies cf. to standard CASU output. This is fine as is but we will keep an eye on the action particularly if it involves unpicking the sky subtraction and stacking and rebadging products. There were a few emails about extinction coefficients that didn't quite make sense but as we are going to change all this in the next version release this is not a major concern. The short story is that for Rx=Ax/E(B-V) for VISTA filters we think WFAU are using Rj and Rk 0.928 and 0.388 [*E'(B-V)] whereas we currently use the UKIRT values 0.902 and 0.367. But as all this will change when we ditch the Bonifacio et al. E(B-V) modification and adopt new Rx values more akin to (but not quite) the Schafly et al. values based on the VVV Bulge and disk results, it is on hold. Still no new phase3 news, which must still be new good news. MJI has been attempting to help out the Montage folk deal with a ticket based on VISTA tile images. However, as all tests point to Montage being the problem, e.g. ds9 and fverify are happy, it does not appear to be an issue with the VISTA data. An unusual request for early access to some recently taken VIKING (20130903) data was received a few weeks ago from Alastair Edge (needed for an upcoming ESO observing run). Unusual in that we don't get these requests for VISTA data very often and even more so when the files asked for were really a.) a different night to requested and b.) different run numbers. Fortunately the actual night (20130902) had been processed and the coordinates were also supplied which help resolved the mystery of which run numbers. Finally, just to prove no matter how clear we think the technical web pages are on certain issues, a query regarding units of fluxes in images and catalogs (ADUs or e-) came in. Sure enough, even though we thought it obvious it was hard to find the information. STH and CGF will illuminate. <<<< 8. VST update -------------- Processed data to end of September fully calibrated and released. Further processing on hold until the quarterly band-merged updates have been completed. Talking of calibration, the VPHAS PI asked if we apply the offset (3.09) to H-alpha from the r-band ZP in the postage stamps served up by the website. Nope, the H-alpha ZP would only be optionally adjusted at the band-merging stage. The ATLAS PI had yet more VST phase3 query feedback from ESO mainly to do with numerology. Fortunately EGS had kept a copy of the phase3 deliverables (well almost, it did include a bunch of undelivered "test depth" images and conf maps too). We assumed the other 10677 catalogues and associated images and conf maps therein were as delivered to ESO giving a breakdown in bands: u 1662 g 1694 r 1975 i 2506 z 2840 = 10677 total The other relevant information was that we delivered everything, even repeated OBs, and as alway we caution the ESO supplied OB grade/completed keywords are a.) not alway supplied and b.) not always accurate. The ever eagle-eyed Mike Read had spotted a suspiscious bunch of seeing=-1 image catalogue extensions on z-band ATLAS data from 201212. Although these happen semi-randomly occasionally, these turned out to be caused by a bizarre excess of objects in a (almost) zero-confidence part of #21 but only for stacked ATLAS Z-band data. In turn this turned out to be caused by slightly more scattered light than normal from the central detector masking strips in the z-band flat causing slightly more "zero" confidence region strips near the detector edge. It transpired that all these images (correctly) had av conf (col #58) around 0-1 and a simple mod to the toolkit classifier (to flag these as noise) provided a robust workaround. EGS will investigate how the seeing=-1 QC check information is used as part of our pre-release checks and see why/if this was missed. <<<< The ATLAS PI asked for contributions to an ESO messenger article and we duly put in a bit about ATLAS and MW foreground investigations happening at the IoA/CASU. Finally, Tom (for it is he) sent in a new version of the ATLAS phase3 release document and ATLAS DR1 has now been accepted and indeed released !! 9. Spectroscopic stuff ----------------------- GES: Since the latest release of spectra there has been a deafening silence, which means that JRL and EGS are now able to concentrate on other things. As noted earlier there is a WG10,11 meeting in Florence in November to discuss spectral analysis. Just as in the previous minutes all effort on WEAVE and 4MOST has been spent preparing for yet more discussion/meetings. No further news about MOONS. The main upcoming meeting as noted earlier is the WEAVE SGS/OCS meeting at IAC 24-25th October. 10. ESO in-kind stuff --------------------- HAWKI: JRL, GM, AKY and MJI had a meeting to discuss various extra issues arising from checking HAWKI astrometry, photometry etc... summary below. A v0.5 reduction of the entire 2008-2012 collection of HAWKI has been produced and released to ESO. There are lots of issues that have been uncovered by this reduction and many of these have now been dealt with. JRL will redo the 2012 data with the latest pipeline in preparation for a second full reduction in the New Year. The v0.5 reduction will be retained and used for post-processing pipeline development including photometric calibration, further QC checks and progress web page updates. There are still "philosophical" issues to be worked through with ESO regarding the final data products from the ESO pipeline and these will hopefully be dealt with in a telecon on Friday of this week. One of these relates to the creation of HAWKI tile images, which due to the buttable nature of the detector array requires a different approach to that used for VISTA. The software to do this is in hand but requires agreement from ESO regarding the fine details of the method. GM has been looking at the warnings issued as part of the v0.5 pipeline processing of HAWKI data and generated a summary report of those warnings. As part of the HAWKI checks the usual astrometry checks were run and MJI ruined his eyesight even further by eyeballing all the HAWKI pawprints with astrometry issues picked up by this. After cross-matching those against the pipeline-generated errors and warning logs it was clear there were several (oft repeated) fields that continually failed, including a couple of well-worn standard fields and the whole GOODS-S region. All of these repeatedly failed due to paucity of suitable reference objects. After this multitude of checks the v0.5 data products, all 7TB of them, were copied onto USB disks and shipped to Mark Neeser at ESO. We now need to update the photometry and finish off the other QC checks on v0.5 prior to propagating the diagnostics to the web pages. <<<< There was a minor panic about lack of HAWKI data whilst discussing the HAWKI filter list (from 'FILTER1' or 'FILTER2' keywords) aka Ks J H Y NB2090 NB1060 BrG CH4 H2 NB1190 NB0984. There seemed to be a worrying lack of CH4 science data given a 2009 paper on BDs STH pulled out of a hat, but this was resolved after the meeting. JRL summarised the state-of-play of the other pipelines. VIMOS is the least developed of the 3 pipelines but does now have a first pass science recipe which is now in the testing phase. For VIRCAM the reduction recipes are nearly complete. Some of the infrastructure needs to be modified to unify it with the HAWKI and VIMOS pipelines. However, that work needs to wait until the supporting software is a bit more stable particularly as this may affect the summit pipeline. JRL is in discussions with ESO to change the way the pipeline software will be released. As all three are being developed simultaneously with common CASU modules, it makes sense not to release them in series, but to do more frequent releases of all 3 pipelines together. Workflows: AKY has cunningly negociated her way around the minefield that is Reflex and has successfully reduced some HAWKI data with a workflow. 11. AOB ------- Delay in Gaia launch IoA wine & cheese event Ghosts in Euclid images provided a bit of entertainment but went away, or at least got questioned enough, that our offer of help was deferred. Continuing actions ------------------ STH - compute the ZP offsets for the VISTA calibration for y,z-band and MJI update the extinction calibrator coefficients as necessary and CGF keep on trucking JRL - keep reporting on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights SC - ordering another fibre channel raid array aka apm49_b still ongoing EGS - propagate the HAWKI processing info to the status web pages after New actions ----------- JRL - report back on Florence GES jaunt at next meeting MJI - find more disk space for latest ING archive data CGF - check the updated VVV tiles and pawprints are recorded as v1.3.1 and that the list of same is up to date STH - make sure the flux units issue is clarified on the web pages CGF EGS - will investigate how the seeing=-1 QC check information is used as part of our pre-release checks and see why/if this was missed MJI - finish off creating HAWKI QC info for web pages ingest