CASU meeting Thursday April 28th 11:30-13:00 CASU meeting area. Present: MJI, AKY, EGS, JRL, STH, MR Apologies: NAW, SC, JPE, RGM, JCR Agenda: 1. Actions from last meeting 2. Comments on WFAU minutes 3. Hardware update 4. Meetings and telecons 5. Data archives update 6. Optical/NIR processing 7. WFCAM update 8. VISTA update 9. AOB Minutes ======= 1. Actions from the last meeting -------------------------------- AKY - is still examining the UltraVista image from HJM, with particular emphasis on 1) comparison between artefacts in both images, and 2) comparison between the fluxes of objects <<<< STH - has built a plone web page covering the i-band vs 2MASS colour relations saga, but has not released it yet <<<< AKY - showed us the results of the flatfield behaviour investigation. It looks like a monthly update timescale is about right. There are generally linear trends with small (~1%) changes from month to month, with the odd internal gain change reaching 3-4%. The monthy linearity updates show a similar stability and changes here seem to reflect the accuracy in determining the coefficients. JRL will investigate using a "rolling" master linearity set as an alternative since abrupt <<<< changes in non-linearity are not anticipated without hardware updates. AKY will finish off the remaining metrics for the final report including systematics, rms cf. photon noise and bad pixels <<<< MJI - put an example picture and more information about the MJD variation across a tile on the web pages MJI - added a note warning about QC information derived from tiles cf. to the more detailed QC information available from the pawprints. There was then some discussion prompted by EGS about summarising the main component pawprint QC information (average seeing, ellipticity, sky brightness, magzpts) in the tile header <<<< SC - the apm44 kernel got some TLC and it now communicates with the card and thence the JBOD disks AKY - the recalcitrant home-space-filling-script was fixed and of course has not been used since MJI - generated the February, and first few nights of March, detector zpt offsets plus illum corrs EGS - inquired about the 8 thread uplink limit for ESO PHASE III delivery with the positive result that the number has been increased to 16 MJI - an extra column is just about required and will be added for VISTA illumination correction tables from P87-> JRL - the # images reported on the web page is still wrong for the night of 20101225, aka 25 rather than 600 odd, Jim'll fix it <<<< 2. Comments on WFAU Minutes --------------------------- No comments 3. Hardware Update ------------------ The Supermicro JBOD expansion is finally communicating and has been partitioned as 3 RAID6 arrays (c,d,e at 38,20,20TBs respectively). Some reshuffling and rationalising of the stored WFCAM and VISTA data will take place over the next few weeks as a test of the JBOD and hopefully to tidy up disk space usage. The new fibre-channel raid is all set up internally, but is not yet connected up to a host system. MJI noted that all is not well with the internal system monitoring pages. The APM variant does at least still work and is accessible via a different web server (it apparently consumed all the swap space on the new IoA server hence the change). The Gaia one just flatlines after some injudicious changes carried out by the hombre-with-no-name. SC was volunteered to investigate if it can be ressurrected without too much work. <<<< We need to think about switching over to Ubuntu/Debian for all the Blades at some point (they currently use Fedora). It's not urgent, but then again there is a gap in the VISTA dataflow. <<<< 4. Meetings and Telecons ------------------------ The next VISTA IOT is not expected before ~mid-May as it awaits VISTA data to flow a bit first. MJI took part in a VMC Skypecon and whilst doodling realised that we have switched from ESO P86 to P87. So we need to get ready with P87 progress web pages. JRL volunteered to sort this out. <<<< The ESO in-kind software contribution saga continues to roll. At least ESO+CASU have reached agreement on their part, all it now awaits is for ESO Council to do the same. MJI attended NAM and talked about M31 surveys. He noted that Llandudno was pleasantly sunny and rather quaint, in a traditional British seaside town sort of way, and that doing science again for a change was almost fun. 5. Data archives update ----------------------- All are rockin'n'rolling 6. Optical/NIR processing ------------------------- VST first light was March 26th with ~1 arcsec image quality over the field. The INT WFC processing queue awaits some TLC since too much of same was delivered to the Megacam M31 survey and used up all the monthly quota. 7. WFCAM update --------------- There are a few gaps in data due to either weather or a failed power supply unit but other than that processing, checking and release is up to date. Since the problems with a few frames on the 16th and 17th March, MR put in a check to detect cameras with incomplete headers at ingest time, which also attempts to automatically fix them. These occurrences are logged and it will be instructive to see how often the problem happens. [A quick check carried out whilst editing these minutes suggests intermittent problems with camera #2 occured throughout the period 16th March until 18th April. There have been no occurrences since then]. As ever the usual multiple queries about early access to the processed data were received from vigilant PIs and dealt with, including a Bank Holiday special request. Reprocessing queue: UDS H-band reprocessing for some 50-60 nights of 05B-07B updates for the GPS DR8 release (summarised in more detail below) A few GPS fields have been noted as having non-convergent astrometric solutions. This is a well known occassional problem in very crowded fields and lead to a discussion about doing a more thorough search to see if other examples exist, and improving the nightly checks to minimise future occurrences of this problem. An extra header keyword of the # of 2MASS stars available in the image footprint, in conjunction with the existing keywords, would readily help to pick out these problem cases. <<<< Meanwhile, searching the existing DB for occurrences of widely different # of matching 2MASS stars in the astrometric solution for the 4 detectors should also catch the majority of the cases (and possibly throw up the odd false +ve but ......). <<<< These astrometric updates will also be propogated to the VISTA pipeline particularly since the VVV team have already threatened to send some examples of similar issues. <<<< A modest amount of nebulosity filtering and re-cataloguing for selected GPS DR8 fields is currently being undertaken and these will be re-released shortly. <<<< The investigation into ways of improving the WFCAM calibration in regions of extremely high extinction e.g. E(B-V) > 10 is essentially complete. This turned out to be a more subtle problem than originally anticipated and involved also making the aperture correction computation more robust in addition to improving the way 2MASS sources are selected in such high extinction regions. Although this issue only badly affects around 150 WFCAM GPS pointings, after discussing this with the PI we may recalibrate all the GPS data with the improved method to place it on a consistent footing. As other survey data is not affected by this problem these would be left alone. The UKIRT board meeting is a month away and STH reminded MJI of the upcoming 29th April deadline for the UKIRT board reports to be delivered (STH has to do one too). <<<< 8.VISTA update -------------- Everything up to the end of P86 has now been processed to Version 1.1 and awaits P87 data flow starting. The current schedule from ESO is that VISTA goes back on sky on May 4th with some nights set aside for testing with WJS on hand to help out. EGS noted that there was at least one VVV tile without "tmp" in the provenance keywords (was it nebulised ? or processed manually ?) <<<< MJI asked if anyone has tried the ESO "Feel the Universe at Your Fingertips" Top Astronomy Apps for the iPad - honest ? or indeed if anyone has an iPad ? JPE reported via email that VISTA images appear 7 times and Orion is no 1! EGS gave an update on the saga of the Phase-III deliverables. After much to-ings and fro-ings between different servers, we now have a link delivering data via ftp to ESO at around 1-2 Mbytes/sec per thread (with up to 16 threads available). EGS has now transferred VIDEO, VVV and VHS (up until the end of September 2010). For VHS this included catalogues and images for both tiles and pawprints. The other surveys are delivering only tile products. VMC and Viking deliveries are being done by WFAU, while Ultravista are doing their own thing as always. Our understanding is that the PI's need to write a report and release notes prior to the OPC meeting which will be held during the week of the Cambridge Beer festival (shame !). EGS noted that the Phase-III deliverables we transferred comprise around 1.5 Tbytes per survey (compressed) for VVV and VHS and two parts of llareggub for VIDEO. Next came much discussion of problems that accrue in later deep stacks with the use of pawskymask for sky subtraction. This algorithm is only used for processing of the VIDEO survey. Recall that this strategy takes a single pawprint and builds an object mask from iterative stacks of the dithers used. Because there are relatively few dither positions, in this case 7, faint images below the detection threshold leave residual flux in the sky estimate which starts to show in the deep stacked images - up to 0.14 mag photometric distortion in this case. One solution is not to include the current frame in the sky estimate. Another is to use a deep object mask, built from a deeper catalogue (requires iteration usually). A third option is to use more frames that are close enough in time to build the sky and thus minimize the effect. JRL will carry out a few experiments in conjunction with Dave Bonfield from the VIDEO team to decide which route to follow. Version 1.2 upgrades -------------------- find an acceptable solution to pawskymask viz-a-viz deep stacking record the potential no. of astrometric standards in each image footprint in the headers to aid in diagnosing possible astrometric solution faux pas [rumour has it that a user spotted two VVV fields with very large astrometric errors (20 arcsec) -> more details are required to check this out <<<< ensure the tiles inherit the nightly photometry summary information from the pawprints to remove issues with inconsistent nos. re-run all the tile catalogues with the latest photometric calibration updates which include more accurate footprint selection and better rejection of saturated standards extend illumination correction tables by a column either end put a description of the colour equations and a How To use the photometric information in the headers (cf. MegaCam) 9. CASUHELPs ------------ A couple of thought provoking issues came to light: various email exchanges suggest that inverting the linear colour equations on the web pages may be beyond some astronomers, what to do ? what are the values to use for R_J = A_J/ E(B-V) etc... is a common question, which extinction law should we use ? is the usual reply there is some justified confusion over the WCS coordinate system e.g. is it FK5 (as it says in the primary header) or ICRS circa 2MASS epoch as is used in the extension, bearing in mind that the bulk motion of 2MASS stars to the observation epoch renders calling it ICRS a moot point the usual problem with GAIA overlaying catalogues on images depending on which version of GAIA is used - don't blame us problems with compiling the CPL pipeline software on DARWIN (OSX) have not gone away, though why you might want to do that is another question 10. AOB ------- All has been quiet on the CASUrtools front. The IoA web system serving the APM catalogues currently ne marche pas - turns out to be incompatible executable binaries on the assorted systems in use of the main IoA cluster. The longer term solution may be to move these catalogues to one of the CASU web servers. Continuing Actions: ------------------- AKY - compare Henry's deep stack with our equivalent STH - put 2MASS JHKs -> i-band predictor information on web pages AKY - finish off flat-field properties investigations add a few words of wisdom and -> report JRL - fix the 20101225 # of images faux pas New Actions: ------------ JRL - investigate using a "rolling" master linearity correction All - cogitate viz inheriting pawprint QC provenance info in tile headers SC - see if Gaia monitoring web pages can be resuscitated SC - progress Ubuntu/Debian upgrade for Blades JRL - set up VISTA P87 directories and web page MJI - new header keyword to record the number of potential 2MASS astrometric JRL standards in the detector footprint EGS - check how often the WCS convergence problem seems to have occured for MR VISTA and WFCAM by comparing # of matched stds for each detector MJI - finish off nebulising the select DR8 GPS data MJI - write and send off UKIRT Board report EGS - find out if the VVV tile without "tmp" is a one off MJI - contact VVV team to get examples of astrometrix faux pas