Doctoral Dissertation Research: Examining Sequence of Contextualized Items in Science

University Of Washington, Seattle WA

Investigators

Abstract

Student performance on science tests often is substantially influenced by the context of individual test items. This doctoral dissertation research project will systematically describe the sequence patterns used in science test-item contexts and obtain empirical evidence on how the sequence patterns of contextual information supports or hinders student performance on science tests. Contextual information includes supplemental information, such as a vignette or selected background facts that precede or follow a test item. The project will address three gaps in the literature on contextualized items by providing: a consistent theory-based framework for selecting test-item characteristics to be studied; new knowledge about how student performance is influenced by specific elements or features of the test items; and better understanding of subgroups of students and their performance to advance knowledge about how to achieve consistently equitable testing. In particular, this project will produce both theory and empirical evidence regarding how the sequence of contextual information can be attributed to differential performance between English language learners (ELL) and non-ELL groups. Findings from the ELL and non-ELL comparisons will lead to important improvements in how items are developed to ensure test fairness. As a Doctoral Dissertation Research Improvement award, the project will enable a promising student to establish a strong, independent research career. The project's design and data analyses will be framed by three research questions: (1) How is student performance associated with different dimensions of the sequence of contextual information for science test items? (2) Controlling for different levels of linguistic complexity in contexts, how is the contextual sequence linked to science performance of ELLs and non-ELLs? (3) How do different dimensions of sequence of context influence how students interpret and respond to tasks? The project will include multiple facets of inquiry, including psychometrics, test-item creation, misconceptions, and cognitive diagnostic assessment. The project will generate item-development guidelines based on dimensions of sequence of contextual information. Items resulting from this process will evoke students' stored knowledge relevant to the content and/or process skills being assessed. The newly developed science items then will be field tested with a diverse population of eighth graders. A range of psychometric and statistical procedures will be applied to student test scores. This activity will include both think-aloud and eye-tracking student cognitive interviews, with quantitative and qualitative analyses of the cognitive interview data.