CIF21 DIBBs: EI: Continuous Capture of Metadata for Statistical Data

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

As the research community responds to increasing demands for public access to scientific data, the need for improvement in data documentation has become critical. Accurate and complete metadata is essential for data sharing and for interoperability across different data types. However, the process of describing and documenting scientific data has remained a tedious, manual process even when data collection is fully automated. General purpose statistical packages (SPSS, SAS, Stata, R) are fundamental to research in the social and behavioral sciences, environmental sciences, biomedical research, and many other fields, but these packages lack tools for documenting how data are modified and new variables created. By creating tools to capture data transformations from statistical analysis packages, this project creates efficiencies and reduces the costs of data collection, preparation, and re-use. Two research communities with strong metadata standards and heavy reliance on statistical analysis software (social and behavioral sciences and earth observation sciences) are targeted, but the approach is generalizable to other scientific domains. Automating documentation of data transformations involves three main steps. First, the most common data transformation operators are standardized and mapped to the Validation and Transformation Language (VTL), an emerging independent standard for describing operations on data in detail. Second, software parses command scripts for the most widely used statistics packages and translates data transformation operations into VTL. Third, software tools modify metadata files adhering to existing standards to reflect changes to the data. This approach embeds detailed variable-level provenance information into standard metadata, and makes it available for data discovery services and automated data analysis tools. This award by the Advanced Cyberinfrastructure Division is jointly supported by the NSF Directorate for Biological Sciences (Division of Biological Infrastructure), and the NSF Directorate for Social, Behavioral and Economic Sciences (Division of Social and Economic Sciences).