Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

Clemson University, Clemson SC

Investigators

Abstract

Intellectual Merit A simple, reliable cooperative learning group peer evaluation method is the focus of this project. The project addresses an issue that is significant for the effective use of and wider acceptance of cooperative learning in engineering education. The instrument being designed and tested will not provide the same level of feedback as more complex and difficult- to-use instruments, but its reliability and validity should be comparable or higher than those of the best instruments currently available, and its simplicity will encourage widespread adoption by faculty who are not ready to make the commitment to more ambitious approaches. This project builds from what is known about cooperative learning. A multi-item instrument with good administration has good potential for becoming a successful instrument. The project is working on crucial issues in peer evaluation (inflation of scores, identical scores, and bias) and in reliability (number of raters, number of administrations). Because there is no true measure of the quality of teamwork, the project is using multiple assessment methodologies and seeking concurrence among them. Instrument validity is being established through verbal protocol analysis, behavioral observation, and concurrence with results obtained with other validated instruments. Both test-retest reliability and inter-rater reliability are being assessed. The project is being conducted in a distributive fashion in five participating institutions. There are nine faculty across these institutions, experimenting with different methods of assessment in eleven different courses. The breadth of courses and instructors promises to make the developed method robust. Broader impact This peer evaluation instrument is expected to significantly improve both team-based engineering education and assessment activities. Instructors using it are able to improve the teaching of team skills by giving students formative peer feedback. Engineering programs using it can document that their students have the ability to work in teams. The successful dissemination of the prototype instrument through publications and presentations has been promising. The simplicity, validity, and reliability of this instrument can be expected to speed its adoption at other institutions. The involvement of schools from multiple Engineering Education Coalitions is also likely to accelerate its dissemination. The assessment instrument and its associated rubric and instructions are also of potential interest to a large number of academic programs. The instrument under development is useful anywhere cooperative groups are employed. Extensive testing for validation and reliability evaluation for this tool should also minimize any unintended gender or racial bias.