Using Anchoring Vignettes to Improve Measures of Teaching Practice

Rand Corporation, Santa Monica CA

Investigators

Abstract

The accuracy of self report of teachers' perceptions of their instructional practices has been a limitation in research and evaluation in education. This project is using a mixed methodological approach to address the extent to which the use of vignettes of teachers engaging in six different instructional practices lead to more accurate teacher self-report on their own practices. Researchers from the Rand Corporation and others in partnership with the Bill and Melinda Gates Foundation Measures of Effective Teaching Extension study will work with 220 teachers in grades 4-9. Research questions address the extent to which teachers understand the instructional practices being portrayed at different levels in the vignettes, the extent to which the use of anchoring vignettes improve teacher self-report , the extent to which information in the teachers' ratings of the vignettes can be used to improve the alignment of teachers' self-rating and expertly-trained raters observations of videos of these teachers, and how the vignette-calibrated teacher self-reports correlate with estimates of other outcomes of teaching such as student achievement or ratings of teachers. The research findings inform the research on the conditions of instructional practices and hold promise for effective feedback to teachers. Researchers in this project have identified important instructional practices and have developed expert-vetted vignettes that describe several levels of the quality of the instantiation of these practices. This project leverages the work conducted in the Gates MET project and collects survey data and video of classroom teaching from a sample of those teachers. The teachers rank the teaching as described in the vignettes and this ranking is used to calibrate their self-reported rating of their own teaching from surveys. The analysis of teacher videos is be used to provide another measure of teachers' practices. The researchers use the calibrated survey ratings of teaching to correlate with outcomes of teaching including value-added models of student achievement and student and principal ratings to examine the extent to which the calibrated surveys can be used as robust measures of teaching. Teacher evaluation as a policy area is an increasingly important theme in educational reform. The evaluation methodology that can provide more valid and reliable measures of teaching than simple self-report provides better evidence of the current educational practices. Better evidence of improvements in teaching practices as a result of selected professional development for teachers results in more-informed selection of those offerings. The findings from this study identify ways that robust information about teaching practices can been identified in cost-efficient ways that supplement the use of classroom observations.