Astronomy Data Warehouse
Project Leader: Katherine Manson, University of Melbourne
Description
The project is providing efficient, standardised access to key Australian and international astronomical data collections. ‘Access’ entails the ability to discover and query archives (via standard web services) and to extract data from those archives. This is a fundamental infrastructure requirement of the Australian Virtual Observatory (Aus-VO) community. The key components are:
- Developing an Australian registry to hold web service descriptions and easy to use interfaces to the registry
- Providing efficient, standardised access to key Australian and international astronomical data collections
- Making use of the National Grid to hold (and most importantly manage/curate) large astronomy archives and provide infrastructure to make it easy to deploy new archives
Achievements
| 2004 |
- SIA service functional with ATCA Phoenix Deep Field survey image, additional surveys will be added easily
- MACHO data: 90% processed, XML metadata annotations for final 10% in progress
- Remote visualisation service released at ATNF for viewing survey data images (e.g. HIPASS, SGPS,...)
|
| 1H2005 |
- Integration of ATCA online access with the data reduction pipeline for producing quick images of data in the Australia Telescope archive (http://www.atnf.csiro.au/vo/atpl/)
- NVO’s carnivore registry service installed at VPAC - HIPASS catalogue (HICAT) and Pheonix SIAP available through the service
- MACHO dataset available through Aus-VO
- Aus-VO download interfaces for Miriad demonstrated
- Completed assessment of SRB as a useful resource for Aus-VO data warehousing
|
| 2H2005 |
- The software for the Australian registry has been updated to a new version that provides some bug-fixes along with improved efficiency and a more sophisticated search interface
- The ROTSE-IIIa and APT data-streams will be archived automatically in a standard way. The web service that allows user to search for, and retrieve data from, this archive is well under way. Database design for ROTSE/APT archive finalised
- Provided SIA for ATNF image archives and develop generic HTML frontend.
Finalised work on AusVO download
- Documented assessment of data brokers (e.g. SRB) as useful resource for Aus-VO data warehousing
- Reviewed extant SIA/SSA web-service frameworks for deployment on APAC National Grid.
Copied selected archive (e.g. HIPASS) to APAC MDSS
- Automated archival s/w for ROTSE/APT functioning
- Disaster recovery plan in place for ROSET/APT archive
|
Plan and Milestones for 2006
Work in 2006 will focus on the development of a production data query and access service that is stable, reliable, and easily extensible. This system is designed so that new data sets may be added easily and new functionality incorporated without changing the original design or code.
| April |
- Build, test and implement basic, VO-compliant web application for simple cone searches of DB (by sky location and time period) that allows mass downloading of images
- Ensure initial, basic web application is VO compliant
|
| May |
- Normalised conceptual data model to store astronomy metadata
|
| June |
- Physical database design based on datamodel and efficiency requirements
|
| August |
- Web service to query metadata functioning, and complying with the IVOA’s generic data access model
- Research and purchase production server for Phase 2 V.O.-node application. Install required hardware at ac3 on UNSW campus. Deploy current version of archive database and web application in ac3 facilities
|
| September |
- Web service to extract data files from SRB based on results from metadata queries
- Investigate and initiate the implementation of any middleware that will improve efficiency and performance of final Phase 2 web application
|
| October |
- Data Access web service to be deployed for testing
|
| December |
- Production release of web service
- Manage data transfer from APT/ROTSE into backup system and archive DB, and on to collaborators by mail
- Build, test and implement final, Phase 2 web application capable of generating tabular data output (VOTable & FITS format) and imaging previews (JPG). Integrate image analysis software into data pipeline (e.g., SExtractor, VOPlot, Aladin) to provide a range of data products for application users
- Research and integrate any additional V.O. freeware/shareware software with V.O.-node web application
|
Participating Organisations
- University of New South Wales
- University of Western Australia
- University of Melbourne
