Applying Multidimensional Item Response Theory Models to Generate an Interconnected Bank of Items for Earth System Science

Michigan State University, East Lansing MI

Investigators

Abstract

This project will investigate the potential for applying high quality test development strategies to assessment of learning in Earth System Science courses taught at the undergraduate level. Assessing student learning at the collegiate level is difficult, a difficulty that stems from the diversity of U.S. colleges and universities and the diversity of curriculum taught within those institutions. The project team will seek to provide college faculty with an assessment resource that will allow them to evaluate learning in their courses through use of a common bank of questions. These questions will be designed to allow meaningful assessment across different institutional settings, from large research institutions to tribal colleges. In addition, different populations of students will be included in the study, from non-­science majors through advanced undergraduates. This work builds upon prior development of a set of geoscience-­focused questions, some of which were developed in collaboration with tribal college colleagues. This project will provide: 1) a bank of test questions that faculty from diverse institutions can use to evaluate student learning, providing a mechanism for truly understanding if and when learning is occurring; 2) a model for assessment instrument development and analysis that can be followed by colleagues within other science and engineering fields; and 3) training for faculty, particularly tribal college faculty, for development and use of assessment questions that are meaningful within their own institutional and course contexts. This project will apply high quality psychometric standards to the current quantitative assessment practice utilized in higher education Earth System Science. This work will apply cutting-­edge Multidimensional Item Response Theory (MIRT) techniques to item and scale validation as well as scoring of multiple dimensions of Earth System Science understanding. Data will be collected from freshmen, students enrolled in non-­majors courses, geoscience majors enrolled in schools nationally, and students enrolled in tribal colleges. This broad data collection will allow for accurate estimation of ability measures and use of Differential Item Functioning (DIF) to evaluate potential bias. Providing mechanisms for inter-correlation of concept inventory items is vital to guarantee that measurement of learning outcomes is meaningful across educational contexts and can be effectively used to inform higher education policy and practice.