CAREER: Decision-Aware Learning of Adaptive Probabilistic Models from Limited Supervision

Tufts University, Medford MA

Investigators

Abstract

Predictive modeling can help practitioners in many fields make high-stakes decisions in a data-informed way. For example, given ultrasound images of the heart, can a model detect valve disease well enough to help physicians recommend follow-up care? Using historical data, can a model help budget-conscious public health agencies prioritize which neighborhoods would most benefit from interventions to reduce opioid overdoses? While machine learning has shown some preliminary progress at such tasks, practitioners often lack the ability to train models to achieve their specific goals. Today’s off-the-shelf methods are often constructed to be easy to train, but this can compromise decision quality when choices made for ease are not aligned with stakeholder goals. This project will develop “decision aware” methods that make it possible to train models to directly satisfy stakeholder goals in several health applications. When detecting heart disease, methods will limit the fraction of alerts that can be false. When predicting opioid overdose events, methods will focus on identifying high-risk neighborhoods. New methods will adapt model size automatically to the available data and be designed to work even when there are few expert-labeled training examples. This award will support the cross-disciplinary training of PhD students and provide immersive research experiences to undergraduates at Tufts University. The project team will publish software and reusable educational modules to help others use decision-aware methods. The project will advance the theory and practice of training probabilistic models for consequential decisions across three directions. First, decision-aware learning methods will ensure that training objectives can be matched to the intended decision-making task, not just a proxy that is more convenient for gradient descent. For binary classifiers, the team will use carefully constructed bounds and stochastic average gradient methods to achieve desired constraints on false discovery rates or false positive rates. For spatiotemporal forecasting of opioid overdoses, stochastic smoothing methods allow training models that can suggest where to intervene by prioritizing a top-k subset of high-risk areas. Second, new limited supervision methods will ensure success even when expert-derived labels are scarce by leveraging easier-to-acquire unlabeled data, even if it differs from the labeled data. The project team will benchmark existing semi-supervised and self-supervised methods and develop decision-aware extensions that are robust to uncurated unlabeled data. Finally, the project will develop methods that can adapt the size of decision-aware latent variable models automatically to available training data, eliminating the expensive grid searches needed to select model sizes in common practice. Technical innovations will focus on sparse approximations and amortizations that can scale model size beyond what is possible with off-the-shelf code today. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.