CASU meeting Wednesday 26th June 11:30-13:00 APM Present: MJI, AKY, STH, JRL, GM, EGS, RGM, SC Apologies: MR, NAW, CGF, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- STH - the "radial" extent of diffraction spike information was making it to the web pages even as we spoke'd STH - have produced all the band-merged files for the VVV Bulge and Disk MJI fields computation of the Z- and Y-band offsets the rest is ongoing These will be combined with estimates from the Sgr region data. <<<< JRL - continue to make slow but steady progress in replacing apm5 Sybase EGS server with a little help from their friends <<<< MJI - VST dome flats-v-twilight flats comparison est encore bien stalled <<<< AKY - continues to progress VISTA raw data offline backups <<<< CGF - review of general technical pages contents is still ongoing <<<< JRL - vstfix is now in casutools installed in the usual place. There is also a 'hidden' prog called vsttab_a2f which converts the ascii files to FITS, updating the docs on the plone page bien marche MJI - can we retire apm15 yet? who knows ..... but MJI will find out <<<< JRL - the imcore_list radian-v-degree bug has been put to rest - honest GM - finished investigating non-linearity and gain measures for HAWKI STH - sent an example set of FITS headers/info location to Warwick folk SC - even brought the labelling device to the meeting to ensure compliance SC - has set in train ordering another HBA card to bring apm49 in the fold EGS - has made the changes that happen in recent updates to casutools more readily visible on the web pages NB: progress on ALL papers - was meant to occur mid-meetings but got sidetracked this month by MJI's exam duties - next meeting 17th July 2. WFAU minutes --------------- It seems WFAU are still copying v1.3 data and are now on P88. 3. Hardware update (and software) ------------------ MJI asked if there was any fallout from the recent changes to the local DNS servers and SC noted that only the apache server on apm5 need any TLC which would be applied post-ish meeting. <<<< Space on /data/apm44_c, the raw VISTA suppository, is rapidly running out and the simplest option seems to be to transfer earlier raw data (starting from mid-2011) to one of the recent fibre channel raid acquistions to free up some space. The only minor change required is to let the web pages know where the index files have moved, either via soft links or via knowledge, otherwise its a null game changer. <<<< The storage for the raw data for VST has also been reorganised using the new raid array on apm45. A similar process to the above is occuring, namely older raw data is being transferred to the new raid for longer term storage with soft links being used to minimise changes to the web page interface. The cunning plan is to do the same for VST processed data, aka acquire a new fibre channel raid and plonk it on apm46. The current temporary storage for procesased VST data on apm48 will then be reunited with its correct usage. <<<< MJI noted that CASU had nabbed another rack from the re-arrangement of the old BT telecoms/solarium room. It now needs converting into coffee pods. The old bugbear about shortage of home disk space was raised and the heavy mob duly sent around to encourage compliance. 4. Meetings and telecons ------------------------- VMC monthly telecon is tomorrow MJI will try and remember to listen in, partly to find out more about the 2% photometry offset that keeps turning up in emails and WFAU minutes. The UK component of WEAVE will be reviewed at a PPRP meeting on 3rd July. NAW will attend to deal with any Data Management questions arising. STH attended an NGTS planing meeting held at DLR (Berlin) and presented slides showing: (1) Overview of CASU-based approach to processing of NGTS prototype data (telescope on the Geneva observatory), (2) 3rd and 5th order radial distortion terms required for the WCS solution, (3) photometric calibration based on APASS data, (4) production of lightcurves and noise properties of the prototype. STH found that for good data, taken in good conditions, with pixel guiding operating, the data are very close to white-noise limited. STH also spent time discussing CASUrtools with Tom Louden (Warwick, PhD student) which he has used on the same data to produce similar results. STH has attended two CHEOPS telecons with specific attention to the requirements on the quick look pipeline. These are weekly telecons with the next one planned for Thursday 27th June. It is expected that some documentation will need to be produced by the end of July (in draft form). ADASS attendance and abstract submissions (deadline 1st July) were discussed. STH was put off by Google telling him that ADASS was in Harrogate so he won't be going then, but those in the know, currently JRL, MJI, NAW and EGS thought the H in the location was somewhere warmer and have registered to attend. Assorted possible abstracts for talks/posters were discussed and NAW, EGS and JRL will chomp them over a bit more before deciding. <<<< 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ All are up to date and burbling away quietly in the background. A query about tracking usage of the ING archive came from Simon Berry via a recent ING Board meeting. We do this sort of tracking anyway so if and when they really want a bunch of Cornish pasties it shall be made so. Meanwhile silence is golden. 6. WFCAM update ---------------- Receipt, ingest and processing running smoothly as usual. The post-processing is lagging a bit behind with April and May requiring the zpt offsets prior to releasing. <<<< Via WFAU there was a query from a user about a mixed bag of sky level header keywords in UKIDSS data, aka SKYLEVEL & SKYNOISE (good) versus CIRMED & CIRBVAR (internal pipeline generated not so good) and not consistently there either. To recap, SKYLEVEL and SKYNOISE are derived from the catalogues and will be present in all of them. These should also be present in any images associated with a catalogue. CIRMED and CIRBVAR are internal pipeline measures which can be used to gauge the average sky level for component frames if skylevel and/or skynoise values are not present, but these CIR things will not always be present and are not computed in quite as robust a manner. 7. VISTA update ---------------- We have just released two more months of VISTA data so all the VISTA data obtained up to 30th of April 2013 are now processed and fits files are available for download from CASU. As noted in the previous minutes the LTO4 tape copying for VVV'ers happened and indeed the tapes have arrived there courtesy of Eamonn Kerrins' luggage. Coincidently, or maybe not, a query then came in about problems VVV folk were having in producing deep stacks of tiles in a couple of test fields in Baade's window region, also taken as part of b293 field, and in another field b201 - both yielding yukky stripey patterns. MJI will investigate further. <<<< EGS has tested the procedure to fix the odd recalcitrant VISTA product on the v20100422_01103_st_tl tile mentioned in the previous minutes and has replaced the incumbent catalogue with the updated version, updated the relevant headers and labelled the tile and the also updated the component stacked pawprint products as CASUVERS= 'vista version 1.31' to distinguish them from the earlier 1.3 versions. A.N.User complained that the QC fits binary tables were not readable with ds9 or skycat. This is true ! Polite suggestions as to what to do with it were sent back in an email. Apparently processed data prodcuts for Laurent Pagani and his 091.C-0795(A) run on VISTA must have made it to him since he has gone very quiet - enfin ! VIDEO object masks and sky subtraction problemos have been the subject of a few emails but JRL has it in hand - well that's what he said. 8. VST update -------------- A slight hiatus in the processing due to other factors intervening for most of June, but things are off and running again and should catch up with incoming within a week or so. April data is ready to be released, or will be as soon as the illumination corrections are computed and applied to the relevant catalogues. Likewise May, though the relative sparsity of same will make computing accurate corrections even more challenging than usual. <<<< An interesting query aka "what are the flux error columns in the catalogue tables" caused some amusement. The polite reply being - they are the 1-sigma errors in the fluxes including: photon noise from source and sky background and any r/o noise to which one could also add the estimate of the error in the sky background level over the aperture. 9. Spectroscopic stuff ----------------------- Gaia-ESO: a great deal of work has gone via JRL into the Giraffe pipeline. This has concentrated on reducing the amount of rebinning being done in order to minimise noise covariance and on making the wavelength calibration more robust. Data for iDR2 are due to be released to the Gaia-ESO community on 22nd July 2013. This will be a complete reduction with the latetst pipelines for all data up to and including the end of June 2013. There is also a significant amount of archival data that will be reduced and released. In addition to iDR2 there it is expected that the first lot of phase 3 spectra will be delivered to ESO at roughly the same time. It will be interesting to see if the ESO validation tool a.) works and b.) validates the files. Certain folk are also curious to see if ESO officially announce Phase3 for Gaia-ESO before the supposed deadline for delivery of the products. MJI encouraged JRL to inquire of this with Magda but he politely declined. WEAVE, 4MOST, MOONS - have been put to bed for a while 10. ESO in-kind stuff --------------------- The final pronouncement for the mid-term review was received on June 20th. The good news is we passed with minor action items and ESO also agreed with our revised timetable for deliverables. One of the issues raised during the meeting was to try out the proposed CASU flatfield algorithm on datasets where the current ESO recipe ne marche pas due to trailed stars etc.. JRL has done this and convincingly demonstrated that our method rocks (we had every confidence) and runs about 100 times faster (it could hardly fail). Meanwhile on the HAWKI front GM has been making steady progress with the detailed analysis of the calibration files for HAWKI and has been focussing recently on the linearity and gain sequences to better characterise them. GM has looked at the ~3,000 raw frames of archival lamp flat and dark sequences taken between Dec 2008 and Apr 2013 and confirmed that all the detectors are linear (at the 1% level or less) up to ~25,000 ADUs. The lack of pickup noise in the detectors was a welcome find. The gains were carefully calculated and to be relatively stable, apart from some sequences taken with narrowband filters. The median gain values are all about 10% higher than those reported by ESO. The gain does not appear to be correlated with pixel values/ADUs. Curiously, about half of the frames could not be used to measure the linearity/gain due to inappropriately short exposure times or low saturation levels set in the threshold-limited integration mode of the detectors. On this topic, we were notified that there is an upcoming HAWK-I special calibration request to observe the same globular cluster in each of the four detectors in order to derive the relative zeropoint offsets => they seem to be having zpt offset problems - d'er! AKY had a fruitful visit to ESO where she learned the fundamentals of using Reflex as well as constructing simple Reflex workflows. Next step is to start using JRL's Hawk-I recipes to try and form a basic Hawk-I workflow. AKY also learnt how to run the Giraffe Flames Reflex workflow that CASU will be beta testing. The beta testing is expected to be focused on a general comparison of the data analysis methods of the CASU and ESO Reflex pipelines, such as any useful additional steps to the ESO Reflex pipeline and possible tweaks to the parameters, as well as suggestions about possible interaction points and plotting to be added to the ESO Reflex pipeline. MJI has a list of the main processing differences between the CASU Gaia-ESO pipeline and the ESO one from JRL and will forward said info to AKY. <<<< 11. AOB ------- July 17th was chosen for the next CASU wine tasting with theme this time Iberian/Italian wine and food. STH volunteered to acquire a genuine (honest) whole leg of jamon serrano from an unlikely source in Birmingham. Meanwhile the winos volunteered to do some sampling of interesting wines for the occasion. Continuing actions ------------------ STH - compute the ZP offsets for the VISTA calibration for z-band and cf. with MJI cunning use of two-colour diagrams in Sgr and Bulge regions JRL - keep reporting on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights AKY - keep applying some TLC to the VISTA raw data tape backups MJI - can we retire apm15 yet ? New actions ----------- SC - apply some TLC to apm5's apaches EGS - figure out easiest way to let web pages know about VISTA raw data moves SC - order yet another fibre channel raid array, this time for apm49 ALL - sort out and submit abstracts for ADASS in time for 1st July deadline MJI - compute zpt offsets for WFCAM may data and apply MJI - investigate yukky stripey patterns in stacked VVV tiles MJI - illumcor time for VST May data and boot April out of the door MJI - forward to AKY JRL's list of gripes (assessment) of ESO's GIRAFFE pipeline