CASU meeting Wednesday 28th November 11:30-13:00 APM Present: MJI, SC, AKY, STH, EGS, JRL, NAW Apologies: MR, JPE, RGM Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. WFCAM update 7. VISTA update 8. VST update 9. Spectroscopic stuff 10. ESO in-kind stuff 11. AOB/news 1. Actions from last meeting ----------------------------- JPE - have characterised the "radial" extent of diffraction spikes for STH VISTA information will go in the technical web pages - eventually <<<< MJI - cf. UCAC4 and PPMXL as alternative astrometric catalogues JRL decided we need the incentive of direct astrometric tests <<<< MJI - a prototype version of a generalised bandmerging program for VPHAS data has been generated. EGS will use this as the front-end for a tailored VPHAS-specific final output stage <<<< SC - is still expediting the handover of the spare LTO4 library unit plus associated 30 TB disk system - nearly there <<<< MJI - still hasn't plonked down random apertures to measure tile noise <<<< MJI - has propagated the processed UVEX to band-merged products MJI - put a note on web pages about pawsky limitations STH - is still updating the ZP offsets for the VISTA calibration paper and showed an update from comparison with overlapping WFCAM J,H,K that suggested the J,H,Ks were behaving as expected, i.e. J and Ks have negligible offset wrt Vega and H the anticipated 3% offset. Updating the Y and Z computations is still ongoing <<<< MJI - reformatted RAID array as two 14TB RAID 6 LUNs, coerced it into rack and thence connected it to replace the earlier apm23_c and bring forth the new apm23_f JRL - progess with retiring our apm5 Sybase server on hold for now <<<< AKY - ensured a P90 folder was created for VISTA processing monitoring EGS - still needs to fix mysterious web page leftovers for 20120809 <<<< EGS - composed a list of ESO OB status blanks for VISTA and VST and sent it to ESO courtesy of Marina MJI - forgot to check if the video board for #85-88 really got replaced <<<< MJI - put a note about VST photometric calibration on web pages MJI - still to do the VST dome flats -v- twilight flats comparison <<<< JRL - notes of limitations/mods of the ESO Giraffe pipeline - ongoing <<<< 2. WFAU minutes --------------- WFAU are indeed copying VISTA files again as of 2 weeks ago up to end of August; and some WFCAM (end of July); and VST up to end of Sept. Apparently our minutes are rather long - we aim to please but admittedly occasionally miss. 3. Hardware update ------------------- The power supply on the library LTO2 unit on apm7 went phffutt. Naturally nobody has these in stock anymore (sic!) but our local supplier managed to get a (almost) new one on e-bay for us and the unit is back up and trucking. Assorted databases have been migrated across to apm49 and some of them still work - apm15's retirement moves ever closer, which just leaves retiring our old sybase server apm5. MJI asked EGS if we could have a wfcamsp/ equivalent to vistasp/ and vstsp/ to help in tracking down files/regions when troubleshooting. <<<< SC and MJI hatched a cunning plan for future proofing the VISTA and VST disk storage for the next few years without a concomitent expansion of CPUs to maintain and will order some HBA FC cards and give it a shot using the now spare RAID array for tests. <<<< A batch of ports on one our G-bit switches started playing up over the previous weekend causing some interesting connectivity problems. SC will investigate either procurring a spare or resurrecting a previous spare. <<<< 4. Meetings and telecons ------------------------- The usual monthly VMC skypecon occured on 22nd November and was mostly about science with no notable data product issues. NAW attended the Science with 4MOST meeting 13-15th November and reported back on the latest status. The 4MOST heavies will be turning up on the 14th December to discuss IoA and CASU involvement in the next phase (if the current one is successful). There was also a WEAVE System Managers meeting in La Palma 21-22 November which we gave up on attending. STH attended the NGTS (Next Generation Transit Survey) meeting in Warwick on November 22nd, and presented a summary of the CASU approach to processing, especially with respect to handling precision time-series photometry. We discussed possible roles for CASU including (i) contributions to modules within the main data management system, (ii) real-time QC monitoring and reporting in Paranal (which could be extended to become a near-real-time Alerts (Transients) System, (iii) development for prototype and commisioning data. From now on STH will attend regular telecons, and has already downloaded some prototype NGTS test data. There is an upcoming KiDS team meeting in Leiden, Wednesday 12 and Thursday 13 December, which was greeted with less than rampant enthusiasm. MJI will check if anyone from ATLAS is attending. <<<< Several of us have signed up for the VISTA meeting at ROE January 17-18th. STH wondered if there were any talk slots still available ? Other upcoming meetings include: the Gaia-ESO survey all-hands meeting Nice 8-11th April and a VMC meeting sometime 8th-12th April in Potsdam. 5. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA ------------------------ A few more UVEX (all of 2010 we hope) have been propagated to the band-merged products stage and the whole lot served through a mike portal for now. One Kepler file with lack of HDU info (kepler_0489o), that failed after the band-merging stage, needed ever more TLC to get it out of the door. 6. WFCAM update ---------------- A combination of poor weather and UKIRT closures (20121114 and 20121115) due to JCMT engineering shutdown lead to a lighter than average WFCAM processing workload this month. Raw data transfers, processing and release of data (to end of October) are all on track. The swapped out old RAID (apm23_c) has now been replaced and this plus its other half (apm23_f) will be used for future WFCAM data storage. There were the usual fast data delivery requests which were dealt with as they came in, including a couple from the out-of-focus folks on Hawaii. The two Mikes had a fascinating email discussion about aperture corrections, and in particular the largest aperture values. This lead 1/2 of them to delve a bit further into the recent WFCAM apcor7's and conclude that since 20120201 a not quite optimum Moffat value has been used in the pipeline classifier. This leads to offsets of ~1-2% in the whole apcor system and corresponding increases in the derived zero-points. The good news is these two effects cancel out such that the ensuing derived magnitudes are correct, the bad news is that we are going to fix them all anyway. 7. VISTA update ---------------- Raw data transfers ingest, tape backups, processing and releases are on track with September just done and dusted. October first pass processing is done. A reminder that the VISTA cryo intervention from December 8th-17th provides a chance to catchup with the various upgrades to catalogue products required to go from v1.2 to 1.3 and to fix a few longstanding minor issues. <<<< JPE forwarded a warning about VIRCAM warming up at around 20:51 UT on the night of 1st November and was probably in this state for much of the night. JRL commented that this would explain the extra set of darks taken later that night. He also noted that his whinge about the paucity of twilight flats at the recent VISTA IOT had a positive impact. The UltraVISTA downloaders whinged in turn about missing skies. However, they turned out not to be missing but simply not all listed in the stacked pawprint keyword (it just inherits the one from the first file in the stacking sequence). Should VISTA stacked pawprints list all skies used ? As its currently only UltraVISTA that uses more than one sky frame within the stacked sequence and as the component files list them anyway this one has been left as is. STH reported on the recent results for the VISTA to Vega zero-point offsets. Comparison with WFCAM data showed that J and Ks behave as expected and have close to 0% offset, while H also behaves as expected and shows a similar 3% offset as that required for WFCAM Unfortunately there is no convenient overlap region in the Y- and Z-bands so he will be revisiting the earlier standard field comparisons to nail them. JPE emailed about a VISTA NB118 daytime test ??, which turned out to mean a twilight time test, investigating red leaks in the NB118 filters. Observations will be of selected very red stars moved to cover each filter in the array to quantify the leakage. Maria-Rosa sent in a query about some strange astrometric offsets of order ~0.2 arcsec that turned up in a few tiles while she was investigating proper motions in VMC tile data. After a bit of investigating it turned out that tiles v20091129_00522, v20091124_00324, v20091118_00412, and v20091113_00082 had (the same !!) residual systematics of order +/-0.2 arcsec across the tile. Redoing the WCS astrometric calibration for these caused the systematic pattern to disappear. The average rms over the whole tile then dropped to ~15mas which is what it is for the later tiles on this field -> unknown cause, simple solution, puzzle as to why since random sampling of VMC tiles after 20091130 all looked fine. This also implies that all the component pawprint astrometric solutions must have been good. As we are going to update the earlier tile catalogues, including these, to fix other minor issues anyway during the VISTA intervention in December (when we update all the tile catalogues to v1.3) there is no point re-releasing these yet. Likewise until the astrometry systematics cause is identified we will defer doing any other astrometry updates on tiles around that date. A mystery item at the 8th VVV Science Team Workshop -Sebastian Gurovich "Comparison between VSA and CASU magnitudes of southern bulge fields" - caught the eye and transpired to be a few% systematics pattern present when comparing single CASU tiles with WFAU deep stacked tiles. It took a while to track down but was caused by grouted tiles pre-20101101 not having detector-level delmagzpts applied i.e. they are not v1.1 despite what it says on the tin - nostra culpa. This must have been the first implementation of the tile grouter that only corrected for seeing/PSF variations. The repeating pattern seen in the comparisons is basically the detector-level magzpt differences in the pre-20101101 single OB tiles. Re-grouting the old v1.0 (presumably) grouted tile catalogues from the originals we keep here fixes the problem. There is a cryptic note on our version log web page on this that is a tad ambiguous in meaning (and even downright misleading) so a further note has been added to spell this out. As we are planning a series of minor upgrades to VISTA tile catalalogues and have to regrout all the tile catalogues anyway in going to v1.3, this will nail it to the perch. To end this section on a more positive note, VVV released an amazingly successful PR image, clearly a good job we weren't too heavily involved. 8. VST update -------------- Raw data transfers, ingest, and processing are all on schedule. The VST October data has been released after the usual illumination correction fixes to catalogues. Still to address are the random gain changes in some detectors (particularly pre-June 2012). Now that the VPHAS bandmerging upgrade is available it is time to dust off attempts at dependent ab-initio fixing of these. Until recently we have been ignoring the occasional random pickup seen on some VST images. However, as in Murphy's Law, when the VPHAS team try to make colourful PR images from single heavily offset frames, colourful pickup noise patterns result. A good example of its properties and variation from frame to frame turned up as the VPHAS team were trying to make a PR mosaic of the Eagle Nebula using three successive frames with the NB_659 filter offset to cover the CCD gaps and the "cross-shaped" vignetting due to the filter. The first of the above has no pickup, the others do but in different locations and with a subtly different pattern. These variations are typical of what we see in other data, though we haven't investigated what fraction of VST frames are affected by pickup noise yet. The interesting thing are its properties: 1. the same pattern is present on all 32 detectors and in the same pixel location [after allowing for reversing of the top half of the array to coalign the readout times] 2. this means the cause lies outside of the individual Fiera controllers 3. the scaling of the pickup pattern varies from detector to detector by about 20-30% but other than that to quite a good approximation its the same pattern in all detectors 4. its not always present but we could easily determine the fraction affected 5. the overall location of the pattern on affected frames is random Based on the above characteristics we were able to write some software that both detects if any pickup is present and also ab-initio removes the pattern quite successfully if any is found. JRL will bundle up the pickup removing software and release via CASUtools <<<< to save us having to run it for PR purposes. Although the pickup noise is essentially removed for stacked frames, VPHAS do not have this luxury so we will investigate including it in the pipeline processing after further trials. Removing pickup would also help the stacking reliability. 9. Spectroscopic stuff ----------------------- Gaia-ESO - the quarterly release of products was slightly delayed due to having to restructure the UVES processed files shipped to us, but happened on November 6th. Clare Worley began work on 1st November on the QC and user interface side and Anna Hourihane began 2 weeks later to deal with Gaia-ESO project management issues. MJI investigated various interpolation methods to resample extracted spectra to a much finer grid (~x4): nearest neighbour; drizzle; linear interplation; Key cubic interpolation (actually convolution), and found unsurprisingly that nearest neighbour and drizzle gave very similar results, as possibly more surprisingly did linear interpolation and Keys (which explicitly conserves flux whereas on a non-linear grid the former does not). The difference between linear and Keys was negligible in the tests so linear it is then for now. 4MOST - much fun was had knocking up an ETC to backup CASU claims of unlikely amazing performance for a 4m MOS system with the inevitable consequence of a likely revision of exposure times up by a factor of ~2 for the LR surveys. Round 6 of simulations is about to happen so this may have been timely, but maybe not popular. WEAVE - the optical PDR was held in Nice on 6th November and by all accounts went well. Assorted quick-look pipeline questions and answers were bandied about, with questions currently on a role. 10. ESO in-kind stuff --------------------- Nothing to report this month other than the good news that JRL is progressing well and backup to speed on all matters cursing-related. 11. AOB ------- The CASU job negotiations were successfully concluded with both candidates signed up and aiming to start 1st April next year. CASU/APM Xmas party organising: drinks - the usual suspects volunteered; food - with only a slight bit of pursuasion the food cooperative was set up. Finally, a first, the Gaia launch date moved forwards from November 2013 to October 2013 prompting a small sweepstake on when it will move back again. Continuing actions ------------------ JPE - put information regarding the radial extent of diffraction spikes for a STH range of values of pawprint NDIT*DITs on web pages MJI - compare UCAC4 and PPMXL as alternative astrometric catalogues for JRL some of the more awkward test datasets SC - expedite the handover of the spare LTO4 library unit plus associated 30 TB disk system MJI - plonk down randomly placed apertures to estimate VISTA tile noise STH - update the ZP offsets for the VISTA calibration paper for next meeting JRL - report on progess with retiring our apm5 Sybase server EGS - fix mysterious leftover v1.2 manifestation for night of 20120809 MJI - check if the video board for #85-88 really did get replaced MJI - add a note about current VST photometric calibration to web pages MJI - are the VST dome flats that stable and how do they cf. to twilights JRL - write up notes of limitations/mods needed for the ESO Giraffe pipeline New actions ----------- EGS - produce a WFCAM equivalent to vistasp and vstsp as a troubleshooting aid EGS - liaise with VPHAS PI on band-merged products contents SC - order some HBA FC cards and test feasibility using spare RAID array SC - investigate acquiring a spare/new G-bit switch to ward off looming muons MJI - check if anyone from ATLAS is attending upcoming VST meeting in Leiden MJI - dust off all the fixes required to go from v1.2 -> v1.3 for all the EGS VISTA catalogues JRL - bundle up the VST pickup removing software and release via CASUtools