Thursday 19th November 2009 APM Meeting tables 11:40-13:00 Present: MJI, RWA, EGS, JRL, JPE(for item 7), MR, SC, STH, RGM Apologies: NAW Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST ..... 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- 1. Actions from the last meeting MJI stage I software release on hold due to other pressures - ongoing <<<< EGS MJI ditto technical paper - ongoing <<<< MJI ditto utility to convert binary tables plus corrections - ongoing <<<< MJI still comparing WFCAM dome flats with twilight sky flats. Low light <<<< level flats have been taken for both WFCAM and VISTA and it will be interesting to see how the low light level linearity compares. JRL is investigating this aspect <<<< SC two high temperature monitoring alarms have been set up in the A/C rooms. They will alert security if either room temperature goes above 30C; who will in turn contact Paul Aslin or MJI as necessary. The temperature sensors on some of the computer kit in those rooms can also be monitored remotely via webpages to relieve, or possibly cause, paranoia EGS made a night of VISTA data available for WFAU slurping purposes MJI contacted WFAU about the mosaic tile file formats and provided a current example for scrutiny and further possible iteration on content MJI duly wrote and sent in a report to the UKIRT Board two weeks ago. It is available on the usual internal web pages. JRL has reduced the August data testing the stability of the proposed new VISTA dome light source with promising results. All we need to do now is get it wired up properly and take some linearity sequences. MJI investigated the wavelength dependence of the astrometric distortion and could find no measureable effect. In doing this he has updated the distortion model coefficients using all the data from 20090927 to 20091027. Residual systematics are now less than 5 mas over the entire array (see later item for more detail) JPE sent the outstanding NB filter transmission curves to MJI via EXCEL spread sheets EGS investigated the stability of repetition of filter position by examining sky frames and confirmed there was a brief spell at the beginning of the PR run where this was an issue. No further occurrences of the filter wheel position being lost in the middle of the night have occurred yet. The SV data is OK possibly because re-datuming was carried out at regular intervals. MJI measured the VISTA SIQ directly from oversampled PSFs constructed from some of the best seeing data on the SMC (20091005). The profiles meet the 50% SIQ criterion but not the 80% one which assumed Gaussian profiles. Real PSFs are closer to Moffat. In this case one with beta=2.5 fits rather well, as it also does for WFCAM data taken in good seeing. With a realistic profile shape the 80% SIQ criterion would be met. JPE said that trial OBs for some of the PS's had already been taken during the SV run JRL noted that he had not yet arranged for a new anti-surge unit for his office nor bought any beer to propel it - ongoing <<<< SC is still figuring out a practical back-up stategy for some of the machines such as apm25, 26, 28 etc. - ongoing <<<< 2. Comments on the WFAU minutes It was noted that WFAU had asked the TERAPIX team to incorporate Rice compression in SExtractor but were politely fobbed off. 3. Recent and upcoming meetings There was a VISTA IOT on 3rd November including some discussion of the recent SV run and what to do next in terms of comparing results. The last SV night was 2nd November. In addition to the main SV programmes some PS 2 hour trial OBs were also completed (VVV, VMC, and a combined UltraVISTA/VIDEO OB). Some of these 'extra' OBs used an SV project ID making identifying them slightly more challenging. All of this data is still embargoed so apart from the SV data not much soul searching on product data quality has yet taken place. The SV meeting at ESO on 25/26 November was confirmed during the IOT and shortly thereafter a provisional agenda for this was drawn up and iterated on. There have also been several Euclid and PLATO telecons recently to discuss among other things data processing and the ground segment design for both projects. 4. Data archives update JRL reiterated his promise to finish off pootling around with the UKIRT CASS raw data archive by this Xmas. All the others, AAT, ING, and WFCAM are motoring along as usual. Meanwhile, we now have enough VISTA data to need a raw data archive for this to help navigate and JRL has set up a password-protected one for internal use. 5. Optical /NIR processing JRL said that he was doing some HawkI processing for RGM but not much else - other than processing VISTA data, writing pipelines, updating ESO deliverables ....... MJI took 2 weeks to FTP over the latest batch of PAndAS MegaCam data for 09B (a mere 250 GB from the CADC thereby demonstrating how good our raw WFCAM transfers are) and has started processing it in the background. 6. WFCAM update WFCAM processing is up to speed with processing commencing within a few days of the data being taken. [Note for JAC: if there are any special requests to deal with data in a particular way the best way to notify us is via a direct email to MJI and MR since we need them more or less at the same time we get the data i.e. within 24 hours of the final bit of data being taken. Although we interrogate the nightly OMP entries we don't always read them. It is often too late by the time we get the weekly summary since some of the data is already processed. This has rarely been a problem but we noticed one example on the 20th Nov which we will follow up post-processing.] MJI reported that there have been some intermittent problems with camera #2 which occassionally went AWOL for large chunks of a night. These (few) nights are currently flagged as 'unprocessed' for the time being and will be addressed during a lull in other work. The nebulous GPS data needed for DR6 was reprocessed and WFAU (and Phil) have been informed. This just leaves the UDS J,H in the queue. 7. VISTA update Data is (almost) regularly arriving about 2 weeks after the observations. Ingest and verification is usually relatively routine and we currently have data up to 4th November on disk. This has all been processed to the pawprint level. Tiling tests are still ongoing using the SV data as a scapegoat for different strategies. MJI has investigated wavelength dependence of the distortion model for the VISTA focal plane using a month of stacked 2MASS residual data from 27th September to 27th October. There are around 1 million stellar measurements to make use of in most passbands albeit taken at a range of sky PAs. Detailed tests projecting the residuals onto the detector grid reveal no measureable wavelength dependence of the distortion model at levels of < 5 mas. Likewise projecting residuals onto a sky grid (i.e. not rotating back to a fixed PA) also show similar results. During these tests the distortion coefficient values were also investigated and were updated (slightly) before producing the final stacked residual maps (PV2_3 42 -> 44 ; PV2_5 -10000 -> -10300). In practice this means that the systematic astrometric error over the whole detector array is better than 5 mas and hence is already about a factor 4 better than the goal. It also implies that doing accurate proper motion, and even parallax, work with VISTA over large areas is feasible and could lead to some interesting nearby cluster/stellar population projects. JRL added a few more words on the VISTA raw data archive he has created. This is searchable by night, run number, filter, sky position etc.. and will remain strictly in house for the moment. Keen as always, STH asked how he could access it and was told that he would need a username and password, though he might want to consider the processed data equivalent that EGS is constructing. EGS noted that he was working on a graphical interface for this which he thinks will be ready in another week. The SV people at ESO sent back favourable comments on the data processing web pages and a few suggestions for discussion at the upcoming SV meeting at ESO. Agenda items will include technical processing details and a preliminary discussion of the science products for the NGC 253 and Orion surveys. EGS and STH will handle the latter and MJI the former. MJI and STH are going to revisit the subject of the illumination corrections, colour equations and assorted other generic odds and ends in time for the meeting. JPE suggested that since the Z filters were made in two different batches then we may need two different colour equations. STH will tell us on Monday if he can see this (he did and the answer was thankfully no). Meanwhile MJI and EGS computed the relative QE of all the detectors using the internal gains from the flats and the measured gains (e-/ADU) of each detector. The results have been posted on the VISTA technical web pages and will be used to update the overall throughput computations in the ETC. <<<< MJI noted that ROE has the first week of SV data and was actioned to check that the rest was ready to send on. <<<< JRL noted that there were several interesting gotchas with the raw data WCS and array position angle which had involved a workaround in the processing pipeline. The PA of the array was not always accurate in the header (at the level of ~1/100th of a degree) and the corresponding rotation matrix in the FITS header was similarly off. This meant that it had been diffcult to use the computed jitter offsets correctly leading to smearing of images by a few pixels near the edges of the field at large -ve declinations. The pipeline workaround now deals with this issue but it would be good to get to the bottom of why it is occuring in the acquisition system. Steven Beard and Dave Terrett are investigating. JRL noted that many observations contained interrupted OBs, whereby the subsequent OB was restarted from where the previous one failed. This was initially a surprise, but then Murphy's law was applied, aka if it is at all possible to restart an OB half-way through it will indeed happen. Its a pity the FITS header information doesn't reflect Mr. Murphy's flexibility. So, as usual, the nightly processing preview script has acquired a few more if's butt's and elsie's. JPE updated us briefly on the latest installment in the PR date and handover saga. MJI said that we had done all we can for the PR colour pictures and have generated far more than are going to be used in the first PR release. Some of these images are quite stunning particularly those of M42 and the Flame nebula in Orion. JRL said he is preparing a new release of the pipeline and an update of the DRLD over the next few days, hopefully in plenty of time for any issues raised to be discussed during the SV meeting at ESO. <<<< [This subsequently reminded us that the Calibration Plan document also needs updating with some urgency.] <<<< In answer to a question from RGM, JPE and MJI said a meeting of the survey PIs would be arranged in Cambridge sometime in the New Year to discuss the processing of, and results from, dry run data. After the PR date and acceptance JPE will talk to ESO about the release of data already taken. When all this has finally happened we will have a better idea of when the Cambridge meeting with the PIs is likely to occur. 8. Any other business SC raised the spectre of increased power consumption and its effect on the University electricity bill. As an example, the APM building has doubled its power consumption in the last two years due mainly to the installation of a lot of computer kit and extra air conditioning systems. MJI has always been strongly in favour of using natural cooling where possible and wanted to progress this now that the University has finally woken up to the huge increase in electricity bills. [Corollary: it turns out we are probably going to be asked to pay the difference in the CASU electricity bill from two years ago out of our grants income making it even more urgent to come up with a greener solution !] The more important task of arranging the CASU Christmas Party fell to STH, SC and DWE who will form a sub-committee to investigate dates and organise a smidge of food and drink. Everyone was asked to make sure that their online calendar files were up to date for December to help with this. JRL said he would only come if there was a chocolate fountain like 2 years back (yuk !). Continuing actions ------------------ EGS add catalogue software and binary table conversion software to MJI software release pages MJI finish off technical paper updates MJI write utility to convert binary tables with all corrections MJI compare WFCAM dome flats with twilight sky flats JRL quit dithering, buy the beer, and enjoy the anti-surge SC is still figuring out a practical back-up stategy for some of the machines such as apm25, 26, 28 etc... New actions ----------- JRL compare WFCAM and VISTA low light level linearity measures MJI update the overall througput figure in ETC using average detector QEs MJI check whether rest of SV data is ready to go to WFAU JRL release next update of pipeline and DRLD JRL update the Calibration Plan document STH DWE SC decide date for Yuletide celebration. STH