Using Scientific Materials to Compare Disaggregated Versus Holistic Ratings

Ohio State University Research Foundation -Do Not Use, Columbus OH

Investigators

Abstract

The National Institutes of Health (NIH), the National Science Foundation (NSF), and other agencies spend billions of dollars each year evaluating scientific proposals. Only the most meritorious should receive taxpayer dollars. But how should these agencies decide which proposals are the most meritorious? There are two general ways to evaluate proposals, candidates, strategies, and nearly any other entity. The first way is called "holistic." Holistic ratings are characterized by a single overall number which reflects one's opinion. The second way is called "disaggregated." Disaggregated ratings are characterized by one rating being given to each of several criteria. The criteria's ratings are then summed or averaged to obtain a single number, which reflects one's opinion. Some research suggests that disaggregated ratings are superior to holistic ones. However in the late 1990s, both NIH and NSF refused to adopt disaggregated ratings in their own proposal evaluation procedures, because no prior research comparing holistic and disaggregated ratings used materials such as scientific proposals. The proposed research will fill that gap. Attendees at a scientific convention will be asked to rate oral presentations in either a holistic or disaggregated fashion. Two holistic panels of five persons each and two disaggregated panels of five persons each will rate every presentation. The correlation between the two holistic panels' ratings of the presentations will be compared to the correlation between the two disaggregated panels' ratings of the presentations. If prior research applies to scientific presentations, then the correlation between the disaggregated panels' ratings should be higher than the correlation between the holistic panels' ratings. This will provide evidence that disaggregated ratings of scientific materials yields evaluations of higher reliability than do holistic ratings. This result should in turn provide guidance to the scientific agencies as to one way in which their evaluation processes might be improved.