Validities of Standards-Based Science Inquiry Assessments: Implementation Study

Sri International, Menlo Park CA

Investigators

Abstract

To gather evidence about large-scale assessment practices, SRI International proposed a research study to NSF to examine the validities of standards-based science inquiry assessment. In response to negotiating questions from the REPP program, SRI restructured the scope of work of the originally proposed study into two phases: Design and Implementation. SRI was funded to conduct the Design Study in July, 1999. Teams of nationally recognized experts studied the validities of science item formats used in three of the nation's most influential large-scale science assessments-NAEP, TIMSS, and New Standards. The research teams conducted three distinct, but coordinated, studies including an Alignment Study, a Cognitive Analysis Study, and an Empirical Study. When completed, the Design Study will integrate methodologies and preliminary findings from the three studies. Results of the Design Study have contributed to the design and methodologies of the proposed Implementation Study. The Implementation Study will permit an intensive analysis of science items and tasks that have been used in large-scale science assessments at the national, as well as the state level; an in-depth examination of the performances of a large and diverse sample of students; and the application of powerful methodologies that were refined during the Design Study. The specific goals of the Implementation Study are to: (1) develop a framework for characterizing the design features of item/task formats that aim to elicit different types of science learning, (2) conduct a study of the alignment of selected assessment questions and exercise formats from existing reference and extended investigation assessments with the National Science Education Standards, (3) conduct analyses of the cognitive demands placed on learners when solving different types of items/tasks, (4) conduct an empirical study of the validities of different types of assessment formats, in particular, the instructional sensitivity of the different assessment formats, and (5) develop prototype designs of multi-level assessment systems that would provide more comprehensive measures of students' science knowledge and inquiry strategies. Benefits that are likely to flow from the Implementation Study will include robust methodologies and procedures to support the conduct of future studies of inquiry-based science assessment. Procedural pitfalls and item and test design considerations will be identified that can guide the research community and state and local education agencies in the design of better validity studies. In addition, we will draw implications of student learning opportunities for the design of science inquiry achievement studies. The Implementation Study will provide confirmatory evidence of what particular items and item/task formats intend to test and student's actual reasoning and explanations of how assessment formats call on different kinds and levels of science knowledge and inquiry. Finally, we intend to integrate these findings to develop guidelines for how educators may combine reference exam results with data from other exam formats to communicate science achievement levels and make policy decisions about professional development, curriculum, and assessment reforms.