RUI: Explaining analytical heterogeneity in ecology and evolutionary biology

Whitman College, Walla Walla WA

Investigators

Abstract

After scientists have gathered data, they often must analyze the data before they can draw conclusions about how the world works. However, recent research indicates that different researchers, because they analyze the data in different ways, can sometimes draw different conclusions from the same data, which could influence how people ultimately act upon those findings. The purpose of this study is to identify the situations in which different researchers may be more or less likely to come to different conclusions when analyzing the same data. To accomplish this goal, the project leaders will provide the same datasets and research questions to large numbers of researchers in ecology and evolutionary biology. At the end of this study, project leaders will hold workshops at major conferences to share their results and to start conversations within the scientific community about how data analysis practices should or should not change. Undergraduates in Biology, Math and Statistics, and Computer Science will gain research skills, and teaching modules for an ecology lab and a statistical modeling class will be generated. To assess the importance of potential drivers of heterogeneity among analysis outcomes, the project will use two data sets--one from avian ecology, one from plant ecology. Potential drivers of analytical heterogeneity, and how they will be studied include: a. Features of datasets: volunteers will be provided alternate versions of the same data and asked to provide different analyses for each version. Versions will differ in the strength of the main effect, the strength of covariate relationships, and the presence of categorical variables that can be used to divide data into subsets. b. Data processing: the last dataset version provided to analysts will be entirely pre-processed, and no additional processing by volunteers will be allowed prior to analysis. c. Researcher domain expertise: For each of the two biological datasets, volunteer analysts will be recruited from a pool of experts in the study system in question, and more broadly from ecologists and evolutionary biologists at large. After volunteer analysts have submitted their analyses, project leaders will conduct meta-analyses to identify the contributions to analytical heterogeneity of the manipulations described above, which will be used to inform discussions of data analysis standards in ecology, evolutionary biology, and other disciplines. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.