Application of Evidence-Centered Design to State Large-Scale Science Assessment

Sri International, Menlo Park CA

Investigators

Abstract

The underlying claim of this proposal is that traditional test development processes have failed to produce science assessments that address high-reasoning levels, are not integrated with learning and formative assessments, and are too costly to implement at scale. It faces the challenge of conceptualizing and developing effective and affordable development processes that result in complex, interactive (e.g., simulations, investigations), valid, and standards-based science tasks that can be incorporated into existing statewide testing systems. In order to address this challenge, the proposal aims at (1) determining ways in which ECD enhances the quality of large-scale, technology-based assessments for middle school grades in key science education strands; (2) implementing resulting procedures in operational state-level test development; (3) evaluating the effectiveness and generalizability of these procedures, and (4) disseminating findings to the assessment community. The proposal builds on previously funded NSF research projects, such as Principled Assessment Designs for Inquiry?s (PADI), a REC-supported online system, which has developed data structures, representational forms, and software tools for the assessment of science inquiry concepts and skills (e.g., inquiry cycles, model-based reasoning) that allows efficient item development. The proposal also integrates the Evidence-Centered Design (ECD) methodology on which PADI is based. The ECD approach relates the learning to be assessed to an evidence model that specifies the student responses and scores serving as ?evidence of proficiency?. Further, the proposal integrates the Pearson Educational Measurement (PEM) test development infrastructure to create inquiry-based science tasks, and align those with the Minnesota Comprehensive Assessments. Specifically, the project aims at Anticipated products include (a) training materials for intended users, (b) ECD-based processes, (c) data structures and authoring interfaces, (d) exemplar design patterns and science tasks, (e) data analyses of large-scale administration of tasks (previously field- and pilot-tested), and (f) publications in scientific journals. Key questions on quality/validity, efficiency/costs, and technological benefits guide both formative and summative evaluation. Dissemination comprises a technical report series, presentations and workshops at professional organizations? annual meetings, and articles in peer-reviewed journals.