ITR: Real-Time Data Aware System for Earth, Oceanographic, and Environmental Applications

University Of California-San Diego Scripps Inst Of Oceanography, La Jolla CA

Investigators

Abstract

The research in this project will focus on dynamically adapting downstream processing and modeling as sensors are added to or removed from a real-time data network. The research will also include work on methods to detect automatically the occurrence of interesting phenomena in real-time data streams and to trigger responses such as data analysis, modeling computations or turning on or off parts of the sensor network. Once developed, this type of capability will be potentially useful to large-scale environmental observing projects such as NEON, Earthscope, NEES and ORION. The project builds on the wireless networking infrastructure developed under the HPWREN project, the data management approach developed under the ROADNet project, and existing ideas about "data grids" such as those being developed and tested under the BIRN and GEON projects. The latter data grid technology is built around the concept of finding and manipulating data after it is stored in databases. The proposed project will investigate how to extend data grid ideas to include real-time data streams and permit feedback between observations and the operation of the observing network. Tasks involved including developing the data cache structure and data flow management needed for a distributed data-handling system for multiple real-time data streams of varying bandwidth, the development of an information discovery system capable of searching across a hierarchical and multi-disciplinary metadata structure, the extraction of metadata from real-time information flows, and the integration of asynchronous, autonomous processing capability at different stages of the data transport system. The work includes merging the existing ORB and SRB technology into a virtual object ring buffer (VORB) and virtual metadata catalogue architecture for real-time data. The planned design for the VORBs also includes the addition of data-push features. The project will produce a working prototype, with data flowing from a variety of different sensor networks including the Virtual Seismic Network, hydrological and meteorological observations from Yosemite National Park and the Santa Margarita River basin, the San Diego Coastal Monitoring and Observing System and the Southern California Coastal Ocean Observing System, a GPS/seismic system for Orange and Western Riverside Counties, and SDSU field stations. The project has the potential for impacts in several areas of environmental research. Once developed, this type of capability will be potentially useful to large-scale environmental observing projects such as NEON, Earthscope, NEES, ORION, the LTER network, environmental engineering field facilities, and hydrologic observing networks. The successful development and demonstration of a technology to integrate and manage archived and real-time data coming from a diverse collection of observing systems could also provide a common framework for integrated access to environmental data collected from observing systems supported by a diverse collection of federal and state agencies.