CASU meeting Wednesday 21st August 11:30-13:00 APM Present: MJI, AKY, STH, JRL, GM, EGS, CGF, SC, NAW Apologies: MR, RGM, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- Agenda item: progress on ALL papers - was meant to occur mid-meets but got sidetracked by holidays and Sesto meeting. STH - the information about the "radial" extent of diffraction spike information did make it to the web pages and impressive it is too http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/spikes-and-halos and is even being used by VVV to help mask off bright star spurions STH - having produced all the band-merged files for the VVV Bulge and Disk MJI fields computation of the Z- and Y-band offsets is still ongoing <<<< JRL - replacing apm5 Sybase server proceeding apace albeit slowly <<<< EGS MJI - VST dome flats-v-twilight flats no progress but watch this space <<<< CGF - review of general technical pages contents going well and dropped as recurring action item MJI - discovered not possible to retire apm15 yet since it is dealing with VISTA processing progress web pages SC - applied some TLC to apm5's apaches EGS - figured out easiest way to let web pages know about VISTA raw data mv MJI - and did the deed by roping in apm47_b for the earlier raw data SC - ordering yet another fibre channel raid for apm49 on hold <<<< MJI - moved the latest VST processed repository from apm48_b to the recently acquired apm46_b its intended home ALL - sorted out and submited abstracts for ADASS in time for deadline NAW will give spectroscopic processing talk and JRL and EGS will also attend MJI - computed and applied zpt offsets for WFCAM April and May data MJI - investigated yukky stripey patterns in stacked VVV tiles and found that user error and/or bad weather conditions were to blame MJI - illumcor corrections for VST April and May data were computed MJI - forwarded JRL's list of gripes (assessment) of ESO's GIRAFFE pipeline to AKY 2. WFAU minutes --------------- WFAU have copied all v1.3 data so must be time to tell them about which files are part of the new v1.3.1 updates (not quite the v1.31 reported in the previous minutes). 3. Hardware update (and software) ------------------ CFG commented that one of the blades aka apm41 was going walkabout and needed some TLC from SC <<<< Space on /data/apm44_c, the raw VISTA transfer disk, was almost out and the simplest solution was to transfer earlier raw VISTA files to apm47_b to minimise changes. The VST raw storage had been previously rationalised onto the two apm45 raids and the recently acquired apm46_b, together with the original apm46_a, are now the VST processed data equivalents. After a few web page hiccups all is now running smoothly. Given all these recent changes STH asked for an update of the APM hardware usage. <<<< 4. Meetings and telecons ------------------------- MJI failed yet again to listen in to the monthly VMC telecon but did at least read the subsequent minutes. No CASU processing issues had been raised. The mid-July Italian -v- Iberian wine and food tasting was well-attended but some of the photos and movies taken subsequently caused both much amusement and some later chagrin. The event proved such a roaring success that the non-CASU postdocs who regularly attend these were sufficiently provoked to return the favour with a mid-August cocktail-based BBQ. WEAVE had its PPRP meeting on 3rd July, which went well and as usual lead to a bunch of questions including the usual suspects. 4MOST has its turn shortly. There is a possibility for CASU to get involved in MOONS data processing but as far as we know at this stage it would be to do some of the higher level analysis (L2) as GEPI appear to be doing the core processing (+ Edinburgh the archive). There is an upcoming EGAPS meeting in Warwick 5-6th September and MJI will attend for the first day, at least, in order to present a VST VPHAS processing and calibration update. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ All the archives are up to date and the UKIRT CASS archive even had a satisfied user who was trawling (incorrectly) for Jupiter data. To do this you look at the 'search method' just above the object name and change the option to "Don't use Name Resolver" as planets have a nasty habit of moving. Then if you use the object name JUPITER (i.e. all upper case) the results come through. IPHAS DR2 should happen soon and EGS has been doing some checks of the catalogues. They have a light version (coordinates and magnitudes) and a full version with a complete set of parameters. Javier Mendez (ING) requested that we mention in the ING Archive web pages that the fits files are RICE-compressed. We do already but clearly not prominently enough. EGS will add this information to the email that users get when requesting data downloads. Javier also asked if we could <<<< change the extensions to fits.fz which we politely will decline but will investigate renaming on-the-fly if users are incapable of doing it. <<<< 6. WFCAM update ---------------- Operations were paused for a August holiday from 3rd-21st which gave us a convenient opportunity to catchup with the backlog of zpt updates and releases. The good news is that UKIRT operations have been extended by STFC for three more months to cover the period up to the end of December. CASU processing will continue as usual at least to year end. Two groups are currently developing full proposals and in parallel with these developments, the University of Hawaii (UH) has apparently agreed to take over the legal ownership of UKIRT and responsibility for the site on Mauna Kea when STFC-funded operations cease. Quite what level of CASU involvement in processing will be required from January 2014 onward is still to be determined. In principle CASU is funded to process/reprocess/provide user support until 31st March 2015 so we do not foresee any problems over that timeframe. We were obviously a bit too keen and assumed semester 13B started on 1st August. Apparently not, its still 13A and we need to update the web pages to reflect this as all data collected until the end of December are under semester 13A. <<<< The leftover GPS astrom issue, aka the presence of PV2_3 and PROJP3 mismatched keyword values for some of the 05A, 05B and 06A data has been fixed. This also involved redoing the astrometry for these frames, so although notionally just FITS header changes this will impact some ingested catalogue astrometry. 7. VISTA update ---------------- We have just released two more months of VISTA data so all the VISTA data obtained up to 30th of April 2013 are now processed and fits files are available for download from CASU (http://casu.ast.cam.ac.uk/vistasp/). VISTA data up to end of July has now been processed. Release is pending on fixing a couple of recalcitrant nights. There has been evermore user confusion over VISTA/VEGA/AB/2MASS systems implying we need to make this ever clearer on the web pages and in particular finish off and publish the VISTA photometric calibration paper. An interesting question about MJD-OBS values in the pawprint stacks arose from a user query. The pawprint stacks inherit the ESO values from the original files for whichever file is used as reference when creating the stacks e.g. potentially relevant keywords are MJD-OBS = 56384.40963239 / Obs start DATE-OBS= '2013-04-02T09:49:52.2313' / Observing date According to ESO documentation MJD-OBS is the modified geocentric Julian date of the start time of the observation i.e. JD - 2400000.5 days. http://www.eso.org/projects/dfs/dfs-shared/web/olas/olas-mjdobs-def.html What is not so clear is what exactly DATE-OBS represents since if you look further down the header of the example above you also find UTC = 35391.000 / 09:49:51.000 UTC at start (sec) which does not quite compute to the same value as the MJD and neither does DATE-OBS. For now we therefore recommend using MJD for more accurate timing bearing in mind that in a stack this is only the MJD of the start time of whatever file was used as the reference. The pipeline processed products alway use the first file in the sequence so this is the correct start time for them. Eckhard noticed a minor faux pas in that in addition to its own night of processed data 20130501 also contained soft links to all of that months data. This was easy to fix but a mystery as to how it occurred. VISTA v1.3.1 has been invoked to accommodate the newly fixed versions of a bunch of weird Ks tile catalogues that turned up on more detailed examination v20120905_00219_st_tl Ks v20120905_00267_st_tl b314 v20120905_00339_st_tl v20120907_00408_st_tl b316 v20120908_00013 v20120908_00049 v20121101_00203 J These all showed weird pawprint patterns in the original catalogues and needed regenerating. Undoubtedly more odds and sods will turn up so consider this a moving feast. The LTO4 tapes for VVV'er duly were written on and sent off. JRL took all the new VIDEO object masks and has reprocessed all the extant VIDEO data. Any newly processed VIDEO data automatically includes these masks. Rather than overwrite the existing processed VIDEO data we are keeping this reprocessed data separate for several reasons not the least of which is that the skies will all have changed and there is no guarantee that other processed VISTA data did not use some of these earlier skies, likewise the calibration will have changed and has possible knock on effects on other data calibration (see apm50_a/vista/video_reproc). Boris Haeussler confirmed that the new masks made a great improvement on the VIDEO data and they are happy now. They request though that these data should be merged with the general release (at the moment this reprocessing has been kept separate) so that they can be downloaded from the archive, transferred to WFAU and released eventually by ESO. However, as noted above we are going to ignore this and find alternative solutions. ESO requested some changes to the Phase3 deliverables for VIDEO DR2 that involved changing some headers. This has now been done and the data has been re-uploaded to their ftp server. The VVV team also got some feedback regarding their delivered catalogue data, with some requests which they forwarded to EGS. These related to VSA-related matters, we think, so VVV were told them to contact VSA about them. The VVV position however was to tell ESO to deal with the data set as it is now and these minor things will be fixed in the next release, whenever that may be. The mysterious ghost writing in some VVV tiles (-1 and other suggestive characters) are caused by scattered in reflections from bright stars, probably off filter edges. 8. VST update -------------- VST data receipt and processing is up to date. Data to end of June was released late July and the usual band-merged files for ATLAS had their quarterly updates ahead of some upcoming 2DF spectroscopic runs. July is pipeline processed but not illumination-corrected yet. There have been a few small changes to OB acceptance, aka the 50% rule, whereby the classification of a concatenation of OBs is flagged as "Completed", if at least 50% of the OBs within a concat are classified with a quality grade "A". This does not affect pipeline processing and probably has little impact on end product QC since the ESO grades should only be considered notional anyway. Some trivial changes were requested for the VPHAS Phase3 deliverables to ESO (change of REFERENC and M_EPOCH keywords) but we agreed with ESO that they will incorporate the changes at their end so that we don't need to upload all the data again just for those. There also seemed to be a mismatch between the number of files uploaded and the number of files in the release description provided by the PI and this also now seems to be solved. The VPHAS+ DR1 data release is even now public via the ESO science archive. The announcement is posted on the Phase 3 main web page at: http://www.eso.org/sci/observing/phase3.html The communication from ESO about the content validation and the email about the successful archival and publication of the VPHAS+ DR1 data via the ESO Science Archive Facility represent the acceptance of the data. ATLAS still has a few problems to negotiate around before they get their data accepted and released. Most of these seem to down to a mismatch/ misunderstanding between the documentation provided by the PI and the actual delivered data. MJI and EGS are helping the PI to sort this out. 9. Spectroscopic stuff ----------------------- The ESO-GAIA update is a long story but the bottom line is that (most of the) data from the internal iDR2 have been ingested and made available (finally). Some more data still coming (archival, fixes, simulations, ....). The iDR2 went out at the end of July. So far a few minor issues have been identified, but nothing major. After several iterations the Phase3 products for the first 6 months of data have gone out to ESO. AKY has been carrying out some tests of the ESO-Reflex workflows for the FLAMES/GIRAFFE and FLAMES/UVES pipelines and has compiled a brief report including JRL's gripes about ESO's GIRAFFE pipeline. WEAVE and 4MOST effort has been mainly spent preparing for assessment meetings and ensuring that CASU remains involved in the relevant processing stages. 10. ESO in-kind stuff --------------------- GM has been compiling a series of measures of HAWKI linearity, gains and generating master calibration files prior to JRL running the whole package of HAWKI data through the v0.5 pipeline. Tests on supposedly problem HAWKI flat sequences that the current ESO pipeline fails miserably (and exquisitely slowly) on showed, as expected, that CASU's method rocks. An upcoming HAWK-I special calibration request to observe the same globular cluster in each of the four detectors in order to derive the relative zeropoint offsets suggests people are having problem with zpt offset problems. We shall see. JRL wrote some brief notes for the HAWKI pipeline release and sent a proto- v0.5 of the HAWKI pipeline to ESO on 16 August (delayed from schedule by a few weeks because of holidays). GM has done a lot of work on assessing the quality of the reductions from this pipeline on four months of data. His observations are being used to upgrade the pipeline so that a v0.5 of a full reduction can be done in time for the planned release to ESO in October. GM has also assembled a chronology of changes to HAWKI data and the instrument and is now in finishing off constructing and documenting a convenient look-up table that provides the detector gain versus time. 11. AOB ------- In the interest of the longevity of the group, MJI encouraged everyone to investigate membership of the new University Sports Centre just across the road on the West Cambridge site. Continuing actions ------------------ STH - compute the ZP offsets for the VISTA calibration for z-band and cf. with MJI cunning use of two-colour diagrams in Sgr and Bulge regions JRL - keep reporting on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights SC - order yet another fibre channel raid array aka apm49_b New actions ----------- SC - investigate and preferably fix apm41's misbehaviour MJI - update APM hardware text file so that STH can plonk it on Plone EGS - ensure Rice-compressed nature of ING archive files is included in emails sent to users for downloads EGS - investigate simple alternatives to bulk renaming of files MR - change 13B web pages for WFCAM to 13A