SCISIPBIO: A data-science approach to evaluating the likelihood of fraud and error in published studies

Northwestern University, Evanston IL

Investigators

Abstract

Scientific literature servers several important roles. Within the sciences it can inform future research, can pave the way toward new discoveries, and guide the future plans of individual scientists and how they spend their own time and careers. Outside of the sciences, scientific literature too serves several roles, such as informing policies or guiding individual judicial decisions. For all of these reasons, maintaining the integrity of the scientific literature is of uttermost importance for scientists, the broad public, and ultimately the public’s perception of individual scientific fields. Yet, identifying non-trustworthy scientific literature even remains difficult for scientists and editors of scientific journals. This project seeks to identify suspicious scientific manuscripts before they are publicized, using a data-scientific approach in which we capture many distinct traits of scientific manuscripts and their content, as well as information about the authors. The outcome of the project will include a programmatic and web-based interface that allowed third parties such as policy makers and scientific journals to scan manuscripts for signs of scientific fraud and error. The project will focus on the biomedical sciences. The system beneath this interface includes 81 distinct databases that have been aggregated, annotated (e.g., with the chemical and biological properties of included genes), and linked through publication metadata (e.g., references, authorship, funding). These data will be matched with a database on fraudulent and erroneous publications (using retractionwatch and a manually curated database). Features of fraudulent and non-fraudulent publications will be conditioned on these databases, with additional features based on network-properties of genes and authors. The project will employ distinct machine learning approaches, such as Gradient Boosting and auto-learners, whose performance will be evaluated out-of-sample. Forth, to improve interpretability, and better understand scientific fraud and error, and possibly improve the robustness of models, the project will regularize and simplify the models to reduce their predictive capabilities to a small set of the information. Lastly, the project will create a REST-based interface that will allow the import from custom manuscripts. The proposed work is unique for conditioning manuscripts on highly distinct properties of manuscripts including content and world-leading training data. This will provide a data-driven tool for policy makers and scientific editors to identify suspicious manuscripts before they enter the published scientific record. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.