Collaborative Research: Building Assessment Capacity in Chemistry Education - The CHemistry Instrument Review and Assessment Library (CHIRAL) Project

Auburn University, Auburn AL

Investigators

Abstract

This project aims to serve the national interest in high-quality undergraduate chemistry education by promoting the adoption of evidence-based tools for assessing instruction. Understanding the impact of an educational initiative on student learning requires data from valid assessment instruments that are aligned to the outcome of interest. This project aims to help chemistry instructors and chemistry education researchers select valid instruments for classroom use. Specifically, the project will develop and host a catalog of assessment instruments used by the chemistry education community. The catalog will be available in a user-friendly format and hosted on the Chemical Education Xchange, an American Chemical Society community resource established in 2012. To further support the selection of high-quality instruments, this catalog will also: 1) provide information about the validity and reliability of data collected using the instrument; 2) develop a glossary of the concepts and methods related to validity and reliability; and 3) include expert reviews regarding the available evidence for each assessment. Through this work, the CHemistry Instrument Review and Assessment Library (CHIRAL) platform will be developed as a central resource for the chemical education community. To meet its goals, the project team will systematically review available literature to identify assessment instruments that are used in chemistry education. A panel of experts will write a summary report about each instrument, to aid users in selecting the most appropriate assessment instrument for a given context. Along with basic metadata, such as the topic and format of an instrument, each instrument in the CHIRAL platform will include an overview of the evidence regarding the instrument's validity and reliability. This information will include results from the developers and from other studies that have used the instrument. Additionally, key principles of validity and reliability included in the summary and overview will be hyperlinked to a dynamic glossary. This glossary will summarize and reference existing quality resources that explain common psychometric terminology and methods in a standardized format that scaffolds from basic to advanced topics. To support the value of the CHIRAL platform to the chemistry education community, users will have multiple ways to provide feedback on its features. The information gathered and presented by the CHIRAL platform is predicted to improve users' decision-making processes in evaluating and implementing educational assessments in chemistry. To measure the extent of this impact, a qualitative evaluation study will be conducted to gauge faculty's perspectives of hypothetical instruments supported with varying levels of evidence for the validity and reliability of their data. Within the chemistry education community, the combination of the catalog, peer-review process, and the glossary have the potential to increase the rigor of the learning assessment process, enhance the value of scholarly work in instrument development, and reduce the time educators and researchers dedicate to locating or developing appropriate assessments for their needs. As a result, this project is expected to enhance both instruction at the classroom level and educational research on student learning in chemistry. This project is supported by the NSF Improving Undergraduate STEM Education Program: Education and Human Resources (IUSE: EHR), which supports research and development projects to improve the effectiveness of STEM education for all students. Through the Engaged Student Learning track, the program supports the creation, exploration, and implementation of promising practices and tools. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.