Panel Study on Transparency and Reproducibility for NCSES Statistics and a Workshop on the Implications of Convergence for Measuring the S&E Workforce and the S&E Enterprise

National Academy Of Sciences, Washington DC

Investigators

Abstract

The National Center for Science and Engineering Statistics (NCSES) is the nation's leading provider of statistical data on U.S. Science & Engineering (S&E), including data on research and development, the S&E workforce, the condition and progress of Science, Technology, Engineering, and Mathematics (STEM) education, and U.S. competitiveness in these areas. NCSES conducts a large number of surveys to measure these characteristics; however, social and technological changes, such as declining survey response rates, constrained budget and staff resources, and alternative sources of data from non-government sources, are making surveys more expensive to conduct and less useful. The Committee on National Statistics (CNSTAT) of the National Academies of Sciences, Engineering, and Medicine will conduct two projects for NCSES to improve the relevance and usefulness of its statistical programs and help it address a changing world. The first project is focused on better measurement of scientific work. Because many of the most pressing questions confronting science and society today -- in fields ranging from cell research and nanotechnology to environmental studies and beyond -- are large and complex problems which are inherently interdisciplinary in character, they require collaborative research using multiple methods and approaches from different disciplines. This has been referred to as convergent research, and it is vital that NCSES be able to accurately measure convergent research to inform science policy about the patterns of this research and how it links to scientific advances. However, measurement of this work is difficult because there is no single accepted definition. To assist NCSES in being able to gather useful data on the extent of convergence in scientific training and research, CNSTAT will convene a workshop to identify and discuss issues in measuring convergence and its implications for the science and engineering (S&E) workforce and the S&E enterprise more broadly. The second project is focused on expanding the kinds of data NCSES uses for its statistical program, to potentially incorporate the use of administrative records and other data sets to produce statistics that are relevant, timely, and of high quality for the science policy and research communities. However, using these data sources requires transparency in the sources, quality of data, and methods used so users are able to assess whether the statistics produced are fit for their intended purposes, and for comparison with results from other agencies and countries. In June, 2017, NSF supported a CNSTAT workshop that examined various aspects of transparency and reproducibility of data products from federal statistical agencies, and it was clear that the current approaches to transparency and reproducibility could be improved. Presentations from other countries highlighted the development of a variety of tools, which could be useful for achieving greater transparency and supporting greater use and reuse of statistical data in the United States. CNSTAT will conduct a consensus study that will provide a thorough assessment of these tools and potential benefits to NCSES, and provide recommendations on developing standards for transparency and reproducibility of NCSES statistics to help data users and maximize the public benefits from NCSES statistical data. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.