Hybrid Approaches to Optimizing Evidence Synthesis via Machine Learning and Crowdsourcing

Northeastern University, Boston MA

Investigators

Linked publications & trials

Abstract

Abstract Systematic reviews constitute the highest quality of evidence and form the cornerstone of evidence-based medicine (EBM). Such reviews now inform everything from national health policy guidelines to bedside care. However, systematic reviews are extremely laborious to produce; researchers can no longer keep pace with the massive amount of evidence now being published. Semi-automation of systematic review production via machine learning (ML) has demonstrated the potential to substantially reduce reviewer workload while maintaining comprehensiveness. However, it is unlikely that machines will fully supplant human reviewers in the near future. Rather, human experts will probably remain in the loop, assisted by automated methods. Methods that exploit the intersection of human workers and ML models in the context of systematic reviews have not been explored at length. Furthermore, we believe there is substantial untapped potential in harnessing distributed crowd-workers to contribute to systematic reviews, and thus economize expert reviewer efforts. This novel avenue has largely been neglected as a means of increasing the efficiency of review production. We propose addressing this gap by developing and evaluating novel, hybrid approaches to generating systematic reviews that jointly incorporate domain experts (systematic reviewers), layperson workers recruited via crowdworking platforms such as Amazon's Mechanical Turk and volunteer citizen scientists, while simultaneously capitalizing on ML models. This innovative, hybrid approach will be the first in-depth exploration of intelligent ML/human systems that aim to reduce the workload in the production of biomedical systematic reviews. Our strong preliminary work demonstrates the promise of this general strategy.