SI2-SSE: SciDB - A Scientific DataManagement System

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

Current relational database systems (RDBMSs) were engineered for the business data processing market, and not for scientific users (e.g, astronomers, physicists, chemists, oceanographers, or earth scientists). By and large, science users either grumble and use RDBMSs or, more often, "roll their own" data management software. Significant projects, such as the Large Hadron Collider (LHC) and the NASA Mission to Planet Earth, have spent millions of dollars on custom software systems, with limited applicability to other projects. As such, after a generation of science applications, there is limited shared data management infrastructure. SciDB is a project focused on building an open-source DBMS focused on the needs of science users. We have developed the requirements of SciDB based on a close collaboration with a number of scientists. Data management features include a nested array data model (rather than the tabular model of RDBMSs) with operations attuned to scientific data, a no-overwrite storage model (allowing interaction with historical results), and support for uncertainty, named versions and provenance information. At this point, there is a distributed team of 17 programmers and scientists working actively on the design and implementation of SciDB, assisted by an advisory committee of 15 scientists. This team is primarily focused on the research issues surrounding the design of SciDB, and most contributors are involved in the project as volunteers or are paid by their individual organizations. A demo of the first working proof-of-concept SciDB prototype was given at the VLDB conference in August 2009, and a first public release of SciDB is planned for September 2010. The purpose of this NSF grant is to enhance SciDB with additional science-oriented features, including time travel, versions, uncertainty and provenance. With NSF?s help, we expect to develop a full-function system by the end of the grant period.