Wednesday 21st April 2010 APM Meeting tables 11:30 - 13:00 Present: JCR, MJI, EGS, JRL, MR, STH, RGM, SC, JPE Apologies: NAW Agenda ======= 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent & upcoming meets 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, VST 6. WFCAM update 7. VISTA update 8. AOB Minutes ======== 1. Actions from the last meeting -------------------------------- EGS stage I software release partially done but still more to do. Will include JRL's standalone catalogue software tarball together with the already included example piece of code for applying all the corrections for the catalogue binary FITS tables. <<<< MJI asked that the stacking code not be included yet since it contains some sneaky tricks that are (honest!) going to be published in the WFCAM and VISTA technical papers. MJI the dome flat comparison is still to be done. <<<< STH updating the Calibration Plan document is progressing well. It will better reflect the actual method in use for photometric calibration for VISTA with: the latest colour equations, Vega zero-point offsets from standard stars, AB magnitude conversions, comparison with WFCAM photometry etc.. There's enough high latitude data to do overlap tests. Should be ready by next meeting. <<<< JPE noted an issue with the VISTA NB980/990 split filter which hasn't been tested on the sky yet. MJI pointed out that currently the filter keyword is only present in the PHU and for this we really need to get filter keywords into the FITS extensions. This issue will be raised at the IOT meeting on Friday. <<<< RGM is still testing the catalogue conversion software to destruction <<<< EGS plonked our png versions of the ESO-released PR images on the ALL CASU web pages. Now we need to write a small amount of text for each picture. EGS will put in links/cross references to the ESO versions. <<<< SC has added EGS's machine to the traffic monitoring web page. MJI noted that since the bottleneck was fixed internal traffic now gets close to Gb/s connectivity MR added the extra odds and ends of the rediscovered missing July WFCAM data to the ESO transfers and MJI made sure the raw data was archived to tape MR has put a break point in the progress web pages denoting the end of "Science Verification" and the beginning of the Dark Ages EGS informed the VISTA survey PI, who downloaded data from the night of 20100107 about the fix, whereby a DFU caused an EFU from detector #11 onwards for a small number of OBs EGS are revisiting the SV data and regenerating all the science products MJI encore une fois. They gave a brief update of progress so far. EGS STH noted the meeting in ESO of the Orion team on May 26th and MJI bet that the other M's would want a similar get together for the NGC253 SV project. EGS is making good progress with the Orion SV data and MJI is re-stacking NGC253 data and cursing the intermittent problems (~30% of the time) with channel#14 detector#6, particularly bad for the J-band which is the main bit left to finish off. 2. WFAU minutes ---------------- MJI was a bit puzzled about the comments regarding tardy 10a WFCAM releases seemingly triggered by a Japanese PI who wanted fast access to his processed WFCAM data taken 15th-19th March. The deal has always been to release pipeline processed WFCAM data directly from CASU when asked by the PI for more urgent access than normal. As it happened the monthly photometry updates for March were being done as the email came in on 15th April. These are obviously not usually included in fast access products. MJI then added that some of the delay in releasing February data was caused by one night having to be completely reprocessed after failing final checks done here. And in case you are wondering, even after the monthly updates are done there is one final set of hurdles the data has to overcome, error-free database ingest here, which takes another day or two depending if problems are found or not. In the interests of clarity regarding CASU software we have cobbled together a timely brief history and commented on one or two other aspects that have cropped up recently. The software modules created by CASU for the pipelines exist in only two forms. First are MJI's Fortran versions which are generally used as standalone applications and which MJI also uses to prototype new ideas. The C versions written by JRL are the same algorithmically but are structured differently as they are meant to be part of a pipeline and exist within the CASU C/Perl pipeline infrastructure (cirdr) in addition to being part of the ESO software deliverables. JRL also directly prototypes pipeline modules, but in C and/or Perl rather than Fortran. The sum total of these C/Perl modules are what is used to pipeline process WFCAM and VISTA data. Back when WFCAM was first starting to rock'n roll out serious amounts of data, WFAU requested a copy of the stacking and mosaicing routines which could be use to do deep stacks and large area mosaics. Jonathan Irwin had already produced standalone C routines using the existing toolkit software development routines as algorithmic templates. These were released and updated as needed to keep pace with functionality and toolkit upgrades for WFCAM processing, together with the catalogue generation software and list-driven photometry code (from within cirdr). The stacking routine was functionally equivalent to what the pipeline did (since the pipeline stacker in turn was developed from the toolkit code) and gave essentially (mainly bar rounding errors) identical results. So, although it was not "the pipeline module", differences were negligible. These stacking and mosaicing routines however cannot be used with VISTA data since among other things they lack recognition of the new (now standard) catalogue FITS table WCS format and have no interpolation options, hence the need for newer versions. Although the cataloguing routines came within the full cirdr environment this was never intended for export since it is a peine dans le cul to build and maintain outside of CASU. So, to make things easier for all involved CASU agreed to shift the stacking, WCS fitting and source extraction software from cirdr into a standalone package (casutools). The advantage is that the processing modules in casutools are identical to those in cirdr. The only difference is that the casutools modules are wrapped in C main routines with a slightly different command line interface from the perl routines that are used in cirdr. Interface issues notwithstanding these standalone routines should yield identical results to the pipeline. Inevitably there will be a few teething problems (e.g. local -v- networked copies of 2MASS) with this slightly different approach but in the longer term it should make future upgrades much simpler. 3. Recent and Upcoming Meetings ------------------------------- The two J's gave VISTA presentations at NAM in Glasgow last week: VISTA Performance - Jim Emerson (Queen Mary, Univ of London) The VISTA science pipeline - James Lewis (Institute of Astronomy) as did a bunch of VISTA survey PIs. There followed a bit of discussion on extra non-VDFS processing for the various PI survey programmes. As far as we know, Terrapix are doing the deep stacking, swarping, cataloguing etc... for Ultra-VISTA and may be requested to do likewise for VIDEO; Astrowise seem to be doing something similar for VIKING. The rest seem to be more or less using the contents straight out of the VDFS processing tin. There is an upcoming VISTA IOT telecon on Friday (23rd) 14:00 - 16:00 which JPE and MJI will take part in, but although MJI asked for significant recollections from STH and JRL about the previous VISTA IOT he drew a blank. A not-the-VDMT meeting took place on 7th April. Some of the issues MJI wanted to raise were deferred but are listed here in case he forgets them namely: updated ToR, membership to better reflect VISTA processing and archive expertise, information access (somewhat eased by ESO web publishing some of the oft-requested information) and ....... d'oh! This triggered an alternative latent memory of a statement MJI made at the affable CASU-WFAU information exchange which took the place of the VDMT, regarding data rates of WFCAM cf. to VISTA. Now updated to include the first 3 months data for 2010, the raw Rice-compressed data volume received from WFCAM was 4.0TB whilst that from VISTA was 4.3TB *. Uncompressed these would be roughly 16TB and 17TB respectively. Since the processed output is 2x the raw volume for WFCAM (blame interleaving) and only about 1.5x for VISTA (blame tiling) both survey systems generate comparable data volume handling requirements. In particular, this means that transport infastructure (e.g. UKlight), computing infrastructure and disk storage requirements are similar too and therefore do not require radically different solutions to those already in place. [* In case anyone is puzzled by this, among other differences, VISTA users tend to do more coadds and use longer integrations.] 4. Data Archive --------------- WFCAM, INT, MegaCam, AAT, UKIRT, VISTA all up to date.. 5. Optical / NIR Processing --------------------------- HAWKI - nothing to report apart from another draft paper by Thomas Preibisch, bets on MM eruption timescale were laid. INT - STH doing some processing for Spanish collaborators and EGS is processing 2009 season IPHAS data MegaCam - odds and ends of PAndAS survey data has been re-massaged Subaru - upcoming data on MJI's to do list VST - que ? 6. WFCAM Update --------------- Processing and raw data ingest is up to date. As almost noted previously, February and March 10a processed data have now passed their final checks and have been flagged as ready to copy. Remaining issues arising from 999 file runs from 12,13 July have been dealt with (there were really 3186 and 3329 files), though MR will double-double check the raw WFCAM archive has the extra frames ingested properly. MJI wrote up the details of the nebulosity filter for the upcoming UKIRT newsletter. EGS noted that he had been using the filter with some success for Planck and Herschel data too. MJI commented that it is quite general purpose for image feature separation based on scale size and that the new filtering option, proper 2D cross-shape cf. to the original (faster) consecutive 1D operations allows even more flexiblity. MJI noted that AJA was leaving the building after 10 years and would be relocating to Gemini. We want to thank him for all his hard work toward making WFCAM the success that it is and for making our job easier by providing an excellent interface for all WFCAM-related issues. We all wish him well in his new enterprise and noted with amusement that he will once again be Paul Hirst's boss. One enterprising PhD student, David Sobral, requested, and received (after a few sanity checks) all the raw WFCAM data for CMP/3. As usual we will be interested to see what he makes of it. MJI and STH briefly summarised their knowledge of the state-of-play of the ongoing parallax measures for T720. The team are on the ball, contacting CASU to get access to the data within minutes of a night being flagged as processed. Must be interesting ! 7. VISTA Update --------------- Ingestion and processing data to pawprint level (version 0.8) up to date and even all done up to 30th March. The raw data disks are a bit slow to arrive at the moment (last one came on Monday 12th), perhaps Marco and Simon's ash clouds are delaying post. EGS then briefly outlined the post-processing stages. After processing we run various extras, which includes astrometry checks, photometric calibration and file Rice-tile compression, visual inspection of QC plots and some sample images. The data is then copied to the final processed data directory and FITS headers ingested in a database for further checks. If everything is ok we then update the version of the fits file, keyword CASUVERS, and create a VERSION_* file for the subdir which flags the night as being ready to copy for WFAU. Regarding data transfers, we have received ~100 direct PI requests for data now, totalling some 4TB of (compressed) processed data. JPE gave an update on the progress of the G-bit fibre to connect Paranal to Antofagasta and noted an upcoming SPIE abstract/paper describing the system and performance. Various transfer tests between Santiago and Garching have been made to tune system performance and further tests including to Cambridge are planned. MJI and JRL have finished developing the standard VISTA filtering and mosaicing software and this is now being implemented in the stage II pipeline, which also includes recataloguing from the tiles etc... It was agreed that it was important to know what PIs need from tile products and check also that they understand what they will get. However, demonstrating the results from the method we are proposing is key here and that is still some way off. Tiling without filtering is an option (e.g. to make colourful PR images) but we suspect that most punters will not find these much use for the majority of their science requirements. For now we are proposing to deal with this latter type of tiling on an adhoc request basis and see how it goes. It was noted that the narrowband Fynbo DDT data, which had an exposure time of 5 minutes per image and 2.5 hours per OB, was hard to deal with because the background sky level varies too much. JRL is doing the best he can but in future exposures should aim at ~1 minute (and are still not in danger of being sky background-limited). JRL has implemented the algorithm to use externally generated sky masks and this is another possibility to try here since the region observed is part of the Ultra-VISTA survey field for which we have already received a tile mask. There was a discussion about how best to deal with the overlaps ("ears") between tiles, which is generally more of a problem for larger area surveys aiming for efficient sky coverage. Mosaicing the overlapping bits of "ears" from adjacent tiles and recataloguing from them is clearly the best solution but is not something that can be implemented in a nightly pipeline as it requires a non-causal solution. Several PIs have been unable to download VISTA data easily from MACS (simple CLI tools like wget and lftp are apparently not standard on MAC distributions ), so EGS has written Python scripts to help. "Short wave depression" is not a new psychosis but a more inventive hardware excuse for the problems with the top part of detector#16. Apparently a week after cooldown and all will be well. Slight problem with this theory, so we've opted for the CASU SWFUD technical term. Until this situation changes the best option will probably be to mask out the top 1/3 of the detector for Z -> J bands when tiling. This will be done by zeroing the confidence map in that region when creating the tiles. [The confidence map does not pick this up automatically as poor since over short time periods e.g. during linearity sequences, from which the basic confidence/bad pixel map is made it behaves generally ok.] In a significant fraction of cases the CRVALS of raw VISTA files are set incorrectly to 0.0 in the FITS headers at the telescope. Unsurprisingly this is a problem for dither stacking. The science pipeline here has had a workaround for this (e.g. check and use RA Dec instead), and other similar problems, but the Paranal and Garching pipelines currently don't, causing lots of "tickets". JRL estimates that this happens several times a night. he has already been in contact with Steven Beard about this but the simulator software does not show this problem on long soak tests making it hard to nail down. JPE will raise the issue at the IOT on Friday. <<<< There were a bunch of fascinating but trivial FITS header issues like EXTNAMEs not making it into the science products. JRL has tracked these down and fixed the problems, not that it affects the products in any sentient way. 8. AOB ------- JPE++ and SC-- may be testing streaming VISTA data directly here (and back) to test for any potential bottlenecks in VISTA data transfer. 5 new BLADES (20 cores) have been acquired and added to the VISTA processing kit and SC is in the process of connecting them up and installing an operating system on them. Not much progress on the ESO in-kind-payment front - don't watch this space. EGS pointed out that apm25 & apm26 system software needs to be upgraded soon since the current Ubuntu software version is reaching the end of its 4 year support cycle. The next Ubuntu release is due the end of the month. We may try it out on JRL's machine first which as usual refuses to do his bidding. MJI noted that the UKIRT board report is due by the beginning of May and he will concoct some suitable fabrications to include. <<<< MJI moaned about lack of disk space on /data/apm29_a and hopes that soon Jim'll fix it ..... oh nearly forgot ...... and the recently introduced minor FUD'd IAMFUDs in the WFCAM catalogues. <<<< Continuing Actions: =================== EGS - finish adding catalogue software to release pages MJI - get around to finishing dome flats comparison STH - finish off last bits of calibration plan document update JRL RGM - test catalogure conversion software ALL - contribute text to go with CASU versions of VISTA PR images EGS - having unbutted finish off the SV data reprocessing in time for MJI end of May chinwag STH New Actions: ============ JPE - raise issue of NB980/990 split filter and FITS headers at IOT EGS - to put in links/cross references to the ESO versions of PR images JPE - raise the issue of absent (actually 0.0) CRVALS at the IOT MJI - write and submit UKIRT Board report JRL - tidy up the detritus on /data/apm29_a JRL - fix recently introduced minor perturbations in IAMFUD in catalogues