"Improved Methodologies for Field Experiments: Maximizing Statistical Power While Promoting Replication."

University Of California-Berkeley, Berkeley CA

Investigators

Abstract

This project proposes new methods for analyzing data from randomized controlled trials (RCTs), which are now increasingly used in some areas of economics and other social sciences. Researchers often want to use an RCT to test multiple different hypotheses. However, carrying out this kind of data analysis requires careful consideration of statistical issues; in particular, there is a chance that using the same data to run multiple different statistical analyses will increase the chance of false positives. As a result, economists conducting RCTs are increasingly specifying pre-analysis plans; that is, they are specifying how they plan to analyze the data before beginning the analysis. These plans, however, do limit researchers' ability to reuse data and learn from the data before beginning the statistical analysis. The research team will develop a potential mechanism for researchers to learn from the data without specifying every hypothesis in advance. They will use concepts from machine-learning as well as probability theory to provide a better framework for how to maximize power within a given RCT experiment. The results could therefore improve data analysis in important ways. The research team has evidence that in practice preanalysis plans include tests for many, even hundreds, of hypotheses. As a result current preanalysis plan designs are likely to be underpowered for many effects of economic interest. The team will extend and apply techniques from biostatistics, including gatekeeping, sequential testing, and FDR control to the design of preanalysis plans. They will also develop a split-sample approach in which researchers conduct exploratory analysis on one part of the data set, and, using estimates from that part of the data combined with theory over likely mechanisms, refine their hypotheses before registering a subset of hypotheses to be tested on the remaining part of the data.