CASU meeting Wednesday 27th November 11:30-13:00 APM Present: MJI, STH, JRL, GM, EGS, CGF, RGM, SC Apologies: MR, NAW, AKY, JPE Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- Agenda item: progress on ALL papers - update meeting planned for next Wednesday. STH - further detailed analysis of VVV Bulge and disk fields is ongoing to MJI examine the extinction over the entire region. Better extinction maps CGF have been generated and analysis of the extinction calibrator coefficients and the zero-point offsets is ongoing <<<< JRL - replacing apm5 Sybase server -> no more progress recently so EGS still ongoing <<<< MJI - VST dome flats-v-twilight flats keep watching this space <<<< SC - fibre channel raid for apm49 est arrivee enfin EGS - propagation of HAWKI QC diagnostics to web pages has been done MJI and now has all the usual summary files etc... JRL - gave us a short summary of the GES meeting held in Florence recently and was pleasantly surprised to find it well worth attending MJI - found more disk space for latest ING archive data CGF - checked the updated VVV tiles and pawprints are recorded as v1.3.1 and that the list of same is up to date STH - are making sure the flux units issue is clarified on the web pages <<<< CGF plus a few other odds and ends. Indeed GM and CGF have been browsing through the webpages looking at the flux question and other issues and suggested that probably best would be to have a FAQ or "For the impatient" page addressing clearly how to derive magnitudes, errors and coordinates etc.. GM and CGF will make it so and report back at the next meeting. <<<< EGS - still investigating how the seeing=-1 QC check information is used as part of our pre-release checks and what happens to it <<<< MJI - finished off creating HAWKI QC info for web pages ingest and EGS - propagated said info to web pages 2. WFAU minutes --------------- After noting WFAU comments about v1.3.1 MJI sent list of affected v1.3.1 files for VISTA and reiterated policy aka pas panique this only affects a few tiles and their component pawprint stacks. The v1.3.1 refers to the fixed products only. Nearly all of these have all been picked up whilst attempting to do science with the VVV data and failing for a mix of reasons: some a complete mystery since when regenerated the problem went away; a few benefited from better calibration due to a recent upgrade to the way we deal with high extinction regions; and the odd one needed the image as well as catalogues regenerating. There will undoubtedly be more coming to light as we attempt more science with this and other data. It is also worth noting here that whilst helping the ATLAS and VPHAS PIs write their P93 reports for the OPC meeting EGS noticed that ESO had sneakily updated a lot of the OB grades for VST. We're in the middle of updating our version of the grades here as we keep getting requests from users for info about blank grades (particularly at report writing time), but as this affects 1000's of files we'll have to think how best to resynchronise. We later checked to see if they have done anything similar for VISTA and of course they have though it affects less files. EGS queried this with ESO and apparently the ESO OB grades/completion status are changed/updated regularly. 3. Hardware update (and software) ------------------ As noted earlier the RAID array for apm49_b finally arrived and will be installed post-meeting. <<<< Firing up rempote NFS-mounted disk queries seems to be somewhat slower than usual. MR wondered if it was related to DNS name server changes but as usual trying to reproduce the problem caused it to vanish. MJI asked about the Blades software updates and apm44-49. Former done, latter still pending. One of the Blades (apm39) is apparently playing up, SC will investigate <<<< There was a brief discussion about network traffic. MJI will forward the address of the network monitoring page to those on-site interested (or not)<<<< 4. Meetings and telecons ------------------------- VMC monthly Monday 1-2pm on 18th November - MJI actually managed to remember to skype in. All is still quiet on the next Phase 3 request front though rumours of the next PSP review in early 2014 percolated through. WEAVE DM meeting at IAC 24th,25th October - NAW and EGS attended and discussed some redefinition of the workpackages. CASU is still responsible for the Quick Look and Core processing workpackages and will maintain the local L0/L1 data repository. IAC will cover the L2 advanced data products and archive systems. NAW is currently developing the preliminary design review material for a meeting that will take place end March 2014. On Friday October 25th JRL, AKY, GM and MJI held a videocon with ESO folk to discuss the delivered HAWKI v0.5 data products. This included a summary of the v0.5 reprocessing covering all data taken from first-science in 2008 up to the end of 2012, and its limitations. A discussion of major issues encountered in reprocessing and the best way to address them ensued. We were hoping to hear about any ESO evaluation of the submitted data disks but that had not quite happened yet. JRL gave a brief update of the status of the CASU modules for implementation in the VIRCAM, VIMOS, and HAWK-I pipelines and suggested a few changes to the SoW schedule (more frequent releases) to better reflect the reality of the pipeline development. The MLA for CHEOPS is being prepared and NAW has been putting in some effort for the CHEOPS Ground Segment review - which is now all completed with a successful review outcome (and our Quick Look WP documents updated /amended to reflect the review board comments). 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA, VST ------------------------ The archives are up to date and (mostly) also backed up on LTO4 tapes. ING Archive - several requests from ING regarding increasing the amount of files that can be requested in one go to 10,000 and hiding the logs of the data obtained less than a year ago were implemented. UVEX processing of earlier seasons data is steadily trucking away. Subaru processing - EGS again involved in processing some deep Subaru data for the usual suspects. Backups - web server is backed up and EGS is backing up all pieces of software and services before the Xmas break [damn good job as it happened] 6. WFCAM update ---------------- A query came in via WFAU about the latest UHS data. The August and September data release was held up due to the need to reprocess several nights of data, several hundred files on each night, where the skyflip recipe had been used in a creative and unexpected way. There was no point in releasing that data until it had been reprocessed. Those months have now been released and October awaits some TLC. <<<< There have been the usual requests for early access to WFCAM processed data which were all dealt with in the usual way. Phil Lucas asked about some ancient astrometry problem affecting small parts of DR89 GPS data. This was sorted several months ago and noted in our August minutes. Phil also warned us that there is a tranche of data requiring nebulising et al. details will be sent before the Christmas break. Assorted emails regarding the transfer of ownership of UKIRT and the likely consquences re: processing and archiving have been doing the rounds. The details are not yet in the public domain and indeed some of them are still being negotiated - but watch this spa ce. Meanwhile plans are afoot for a "final" UKIDSS release but quite when where and how is TBD. 7. VISTA update ---------------- Ingest and processing is back on track. September is ready to be released, just waiting for a rogue tile to be checked by JRL, and October is processed to the point where MJI needs to compute the zeropoint offsets. All still remains quiet on the Phase3 front. WFAU found a mysteriously vanished tile ESO grade which should have been inherited from the first pawprint used in the sequence - but wasn't. This appears to be a one-off so a one-off fix was applied. Apparently it was a pleasure for UltraVISTA to be able to announce the release of the fully-reduced and stacked Y,J,H,Ks and NB118 DR2 UltraVISTA images of the COSMOS field from the first three years of UltraVISTA survey data. 8. VST update -------------- Transfer and ingests have been running ok apart from a few hiccups with transfers which stopped after the ESO server responded with an illuminating "internal error". Processing resumed in earnest after the quarterly band-merging exercise had been completed. As noted in the previous minutes ATLAS DR1 has happened joining VPHAS+ at the DR1 stage. So far no news of further Phase3 requirements. MJI contributed a bunch of gradeology to help (confuse) the issue of the ATLAS and VPHAS PIs suplying their status reports for the November OPC shindig. In a nutshell herewith ESO's latest take on the OB grades: A - all within constraints; OB completed B - mostly within constraints - some constraint ~10% violated; OB completed C - out of constraint - OB set to must repeat D - out of constraint, BUT OB set to completed and will not be repeated unless PI yells loudly - QC grades R and Z do not exist, hmmm .............. R - aborted OB - this will be repeated later Z - unclear status/grade - automatic retrieval of the status/grade returned null value NONE and Empty string are not delivered - yeah right and other grades that don't exist are X - not applicable, which we are told is always associated with M - must repeat After updating the grades (see earlier comments) ATLAS stacked images only VPHAS single images (we didn't have a before for these) Grade ATLAS before after VPHAS A | 8930 9397 3860 B | 3788 4065 3325 C | 3166 3402 3052 D | 134 138 94 R | 260 442 139 X | 1 10 0 Z | 0 15 5 Empty | ~1200 391 403 Meanwhile we have started the long haul of updating all the calibration based on using APASS as well as the nightly std fields. This won't however be completed until the end of the year at the earliest. 9. Spectroscopic stuff ----------------------- Gaia-ESO: is basically lying dormant until December when it will spring back into action. The Florence meeting highlighted a few issues which really require JRL's attention (e.g. background subtraction in emission line regions). There will be a further data release at the end of January 2014 and that will be the last one until probably the end of 2014. WEAVE, 4MOST, MOONS: just as in the previous minutes all effort on WEAVE has been spent preparing for yet more discussion/meetings and submitting abstracts for future meetings; still no further news about MOONS and not good news about 4MOST, at least from a UK-wide perspective. 10. ESO in-kind stuff --------------------- JRL has submitted a revised timetable to ESO setting out a more realistic delivery schedule, on the basis that we are developing all three pipelines in parallel. The current state of the software is as follows: -- vircam: small updates but still needs some work to factor in the casu_ library which awaits the latter being more stable to minimise knock-on effects for the QC part of the pipeline software on the mountain -- hawki: now has a postprocess recipe that mosaics the individual exposures, rather than the stacks. This has made it very memory hungry and one of the things that needs to be done is to rewrite part of casu_imstack so that it does things in a more memory efficient way -- vimos: now has a science recipe and a standard star recipe. Neither of these do defringing, nor photometry yet. The defringing software is basically written, the photometry awaits a decision about how we can link APASS in -- casu: the library is almost complete, but a few tasks such as casu_imstack.c need some work plus a few other odds and ends The data reduction plan and the pipeline description documents have all been updated to reflect the current state of play. In some places things are still a little speculative, but that is now a very small portion of the documentation. A version of all 3 pipelines, the documents and the v0.5 reduction of the vimos data (up to and including 2012) will be sent to ESO at the end of the month. A skim through the VIMOS reduction shows the usual problem with astrometry in sparse fields and will need a similar fix to that required for some HAWKI fields GM has compiled a list of astrometric standard objects that can be used to improve the astrometry of the ~ 450 bad HAWKI fields. The list is derived from VISTA, WFCAM, or VST. There are standards that can be used for all but 6 fields. These are being formatted for general incorporation into the HAWKI pipeline and a sample field has been sent to JRL for testing. 11. AOB ------- Gaia lunch and wine and even sweatshirts IoA wine & cheese event and CASU followup event destroyed ever more neurons but also increased gut bacteria diversity. JRL and STH have taken the lead with organising the CASU Christmas party. CGF is sharpening his jamon knife in anticipation. Continuing actions ------------------ STH - compute the ZP offsets for the VISTA calibration for y,z-band and MJI update the extinction calibrator coefficients as necessary etc.. CGF JRL - keep reporting on further progress with retiring our apm5 Sybase server EGS MJI - are the VST dome flats that stable and how do they cf. to twilights STH - make sure the flux units issue is clarified on the web pages CGF [absorbed into a new action below] EGS - will investigate how the seeing=-1 QC check information is used as part of our pre-release checks and see why/if this was missed New actions ----------- GM - produce a FAQ or "For the impatient" page addressing clearly how CFG to derive magnitudes, errors and coordinates etc. STH MJI - install apm49_b RAID array SC - investigate if any Blades are still playing up (e.g. apm39) MJI - forward address of the network monitoring page to those interested MJI - release October WFCAM data