Modeling and Inference for Long-Term Effects

New York University, New York NY

Investigators

Abstract

Recent advancements in data-driven decision-making have transformed fields such as healthcare and digital marketing. These areas now prioritize understanding the long-term consequences of actions and policies. For example, drug design aims to mitigate long-term side effects without compromising immediate efficacy; online recommender systems like Netflix aim to improve long-term customer benefits while balancing them with short-term engagement metrics. Evaluating long-term effects is challenging due to the dynamic nature of environments and the cumulative uncertainty of future predictions, especially in real-life scenarios that require prompt decisions. This grant supports research to develop innovative statistical inference methods and models for estimating long-term effects efficiently and effectively. The PI will integrate research and education by involving graduate students in the research and incorporating findings into mini-courses at workshops. The project will also provide mentoring and support for URM graduate students and postdocs, fostering a diverse and inclusive research community. In more detail, this project proposes several research thrusts that provide various models to capture long-term effects in real-life scenarios. The first thrust focuses on environments with time-homogeneous transitions, assuming a Markovian framework. The main goal is to use system observations to understand dynamics, establish robust estimators, and quantify uncertainty. The methods are expected to handle distributional shifts in data, misspecification in function approximation, and the potential high-dimensionality in models. The second thrust concerns non-stationary dynamic systems. Challenges include determining the change points as the system evolves, selecting the most useful and related data, and using an appropriate surrogate index approach to form a valid estimate. On top of the first two thrusts, the third one involves integrating multiple datasets to facilitate estimation. The goal is to develop methods that combine relevant but non-identical data sources effectively to mitigate the issue of data scarcity. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.