Doctoral Dissertation Research: The File Drawer Problem

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Scientific progress depends on unimpeded information sharing between scientists, and from scientists to the public. Scientific problems, however, are typically ambiguously defined, involve perverse institutional incentives, and promote competition between scientists to come to conclusions before others. As a result of these features, unexpected and unwanted results are common and may not always be reported. If information is not shared, scientists may be more likely to repeat one another's mistakes, reach false conclusions, and be overconfident in their conclusions. This dissertation combines analysis and experimentation to examine the reasoning processes underlying decisions to share information, the usefulness of information shared, and the social costs of failure to properly share information. Possible solutions to these problems will be tested in a field trial. Six experiments will use lay participants in simulated science environments to examine the effects of hindsight, institutional incentives, ambiguity, and competition on their scientific reasoning and decisions about sharing information. In the first two experiments, lay participants will be given descriptions and results of experiments and must evaluate the quality of the research and whether the results should be reported. Some participants will be asked to evaluate the results in foresight, whereas others will be asked to evaluate the results in hindsight. We hypothesize that participants will conclude that hypothetical experiments are in error and should not be reported when they disconfirm the participants? prior beliefs, and that this tendency will be worse in hindsight than in foresight. In a second set of experiments, participants will address three challenges that scientists commonly face: perverse incentives, ambiguity, and competition. When reaching a conclusion can lead to a financial payoff, it is likely that participants will interpret disconfirming evidence as error and decide not to report this evidence. Ambiguity and competition are likely to enhance this perverse effect of incentives. In a final experiment, the trial history of participants from these first three studies will be shared with a new participant who will try to solve the same rule. With these experiments it is possible to examine the reasoning and potential social cost of biased information sharing.