Exploiting Virtual Observatory and Information Technology: Techniques for Astronomy
Nicholas Walton
AstroGrid Project Scientist

Institute of Astronomy,
The University of Cambridge

Lecture #6 Goal:

VO Workflows

Science Usage

Summary: Lecture #6

Review of VO Server Side Capabilities

Source catalogues created with the NVO

Scripting with Aladin

GAVO and Theory

The AstroGrid system

Examples of Science Usage

Redshift determinations

Mining for brown dwarfs

Solar flares: movie creation

All lectures plus supporting material at:

http://www.ast.cam.ac.uk/~naw/VO-Course

Recall: server side/ client side ...

VO Server Side Applications

A number of server based applications are now available which conform to VO standards

These applications are aimed at large processing jobs

Some require user to remain 'logged in'

Asynchronous 'workflow' systems (e.g. AstroGrid) allow complex flows of jobs to be run, capture of 'state' allows user to return to collect results

NVO: WESIX

Not currently a 'true' workflow example ... but hey ...

For a single image

Create the object catalogue

Cross correlate with a number of major catalogue sources

Return results and plot

Try this at: http://frank.phyast.pitt.edu:8080/wsext/

Aladin: Scripting: http://aladin.u-strasbg.fr/java/FAQ.htx#ToC21

Aladin has a scripting capability

Wide range of commands available

Possible to create PERL scripts controlling Aladin

GAVO: theory services

Access to simulations

Main limitation – limited data and non-standard meta-data

Try it at http://www.g-vo.org/mpasims

GAVO: Cluster Finder

Cluster finding algorithm: from Rosat RASS and SDSS catalogues

Compute intensive process

Runs on the GAVO grid: http://gavo3.aip.de:8080/GAVOGRID/

http://www.g-vo.org/clusterfinder/

Note: under development

AstroGrid

Links distributed applications and data

System based on the idea of creating workflows

string sets of processes together

carry out the processing steps remotely

view the results locally

The AstroGrid Testbed

10 Gbit/s

AG

node

AstroGrid: Helper Applications

Range of externally provided applications: tools to analyse and visualise end and intermediate data products

VOPlot: handles VOTable data

VOPlot3D: handles VOTable data – 3D visualisation

Topcat: tabular data and manipulation

Treeview: ability to browse MySpace

VOSpec: spectral plotting and analysis package

Specview: spectral plotting and analysis package

Aladin: data visualisation and catalogue access

MySpace browser capability (read-only)

But first, a quick word about DataScope,

and why we need workflows ...

NVO: DataScope

DataScope Internals: http://heasarc.gsfc.nasa.gov/cgi-bin/vo/datascope/init.pl

User entry of target position/name and search radius

DataScope queries the registry to determine availability of resources

Datascope queries them

Returns results

Results ordered

Image archives (SIAP services)

Catalogues (cone searches)

Observation catalogues

Object catalogues

List of resources that had no information or where unavailable

Click through links to many of the returned results

Possible to download results as a tar file

AstroGrid: AstroScope

AstroGrid science service

Release date 30 June 2005 – check it out then

Key additions to the DataScope

Returns info from UK data providers, e.g. CASU at the IoA

Importantly – allows download of results to AstroGrid's Myspace – hence results will be available for incorporation into AstroGrid workflows

Full details (currently under development) at http://wiki.astrogrid.org/bin/view/Astrogrid/AstroScope

Photometric Redshifts – an Extragalactic Case

Extragalactic Case Workflow

See: http://wiki.astrogrid.org/bin/view/Astrogrid/AgDemoDec2004Extragalactic

AstroGrid Redshift Science Service

Packaged workflow

Enter RA, Dec, radius

System returns objects and redshifts

User Options

Input data (INT-WFS, SDSS)

Redshift apps (hyperz, bpz)

Defaults

Source extraction double image mode (r' image ref)

Plus lots of other sensible default configs for the cross match, the redshift apps etc.

See - http://wiki.astrogrid.org/bin/view/Astrogrid/RedshiftMaker

Mining for Brown Dwarfs – a Galactic Case

Accessing & processing distributed data

Select INT-WFS Observing Log entries for Pleiades in i and z bands with small photometric and pointing errors

Cross-match to get i and z observations of same fields

Extract Zero-point, Seeing, Exposure time from Logs

Construct image URLs for Simple Image Access server at Cambridge

Feed images to SExtractor hosted at JBO

Cone search 2MASS catalogue at ROE for Pleiades region

Cross-match lists of extracted i and z sources and 2MASS sources, all held in MySpace at Leicester

Use TopCat tool to access MySpace files and make colours

Colour-Cutter Tool

An AstroGrid packaged science service

Release date 30 June 2005

http://wiki.astrogrid.org/bin/view/Astrogrid/ColourCutter

User selects a position, radius on the sky and inputs a colour cut selection (e.g. r' – i' > 1.0 )

Service returns list of objects meeting the colour cut critera

Workflow searchs image data sets, generates object catalogues for the relevant image files, and performs the source selections based on the object colours

Solar Moviemaker – a Solar Physics Case

Science Case: Movie of solar energetic events

Obtain a series of movies of solar energetic events (e.g. solar flares).

Solar energetic events, for example flares, coronal mass ejections, are routinely identified and event catalogues are compiled.

Event info in catalogues: start and end time of event, magnitude, location on the solar disk, Active Region number

Currently to study a set of energetic events:

for each event in list, manually place a request for data to archive

Download data, calibrate images (SolarSoft libraries), generate movie

Solar Event Catalogue

This catalogue provides links to data resources associated with 'solar events'

The catalogue is available for interactive use at:http://sec.ts.astro.it/sec_ui.php

The catalogue is queried remotely in this example solar workflow.

Dataset: TRACE

TRACE is a high resolution (0.5”), high cadence (20s) EUV solar imager.

SURF TRACE archive at MSSL – hourly files with >100 images

Movie Maker Workflow

Step A: Query event catalogue

obtain VOTable of solar energetic events via query to EGSO/SEC

Query can be over active region number, magnitude of event

Example query:

select * from sgas_event AS s where s.nar>9850 and s.nar<9880 and s.xray_class>'M5

Start loop over events

Movie Maker Workflow-contd

For each event:

Query TRACE database

start time and event time of solar event obtained from SEC query written into query file for TRACE dataset:

SELECT * FROM fits_trace AS T1 where T1.Keywords/img_time >= \'${refstarttime}\'

and T1.Keywords/img_time < \'${refendtime}\'

Start and end time are now workflow variables

Movie Maker Tool

TRACE files are presented to the movie-maker tool as a series of URLs. CEA downloads URLs as temporary files local to the movie maker tool

IDL Solarsoft – a multi-instrument IDL library for analysis of solar images

Shell script generates IDL Solarsoft session

Solarsoft: read images, select wavelength and apply calibration routines

One MPEG per hourly file generated – concatenate hourly MPEGs into 1 movie

TRACE Field of View

Trace FoV

Solar Movie: WorkFlow

Solar WorkFlow: Log File

Solar Movie Maker

AstroGrid Science Service

Solar movie maker

Prepackaged workflow

User enters NOAA event number or time interval of interest

Service returns movie of that event/ time

Options

Data sets (SOHO EIT/ TRACE/ CDS)

Time range or event

Details at http://wiki.astrogrid.org/bin/view/Astrogrid/MovieMaker

AstroGrid Beta Testing

1st functional release: 31 March 2005

At this stage usernames/ passwords will be available – try out the science services and workflows discussed in this lecture

Beta testers invited

Details linked from http://wiki.astrogrid.org/bin/view/Astrogrid/AgScience

For further information on AstroGrid or any other issues raised in these lectures, please contact me for further details at naw@ast.cam.ac.uk or Room H37 or tel 37503

Lecture 6: Acknowledgements + Refs

Refs as noted on the slides

All lecture slides plus supporting material at:

http://www.ast.cam.ac.uk/~naw/VO-Course

Next Lecture:
The Radio/ sub-mm domain – data challenges (John Richer)

vo-lec05-6

VO course - IoA - Feb/Mar 2005 - Lecture 6: Nic Walton

Nicholas Walton

2004-11-10T15:41:41