III: Medium: Collaborative Research: Counterfactual Learning and Evaluation for Interactive Information Systems

Cornell University, Ithaca NY

Investigators

Abstract

Many information systems engage with their users through the following loop of interactions: the system receives a context as input (e.g. query, user profile), responds with a context-dependent action (e.g. ranking, recommendation, ad), and then receives some explicit or implicit feedback on the quality of the action (e.g. star rating, following a search result, clicking on an ad). While ubiquitous and plentiful, log data from this interaction loop does not fit the standard mold of supervised learning, since the feedback is both biased and partial -- the system determines through its actions where it gets feedback, and even for the chosen actions it typically doesn't observe all feedback (e.g. missing clicks on relevant results in ranking). This project will address the question of how this logged data can nevertheless be used for evaluating and learning new systems. The potential upsides of reusing the existing log data are evident. For evaluation, the use of historic log data enables engineers to rapidly evaluate many new systems offline (e.g. new ranking functions, recommendation policies), without the weeks of delay and the potential negative impact on user experience implied by online A/B testing. For learning, it similarly enables offline reuse of existing data instead of slowly collecting new data through an online learning algorithm. This can greatly speed up the machine-learning development cycle, since model selection, feature selection, and eventual quality control can happen offline before any learned policy gets deployed to the users. Reusing existing log data is particularly important for small-scale information systems (e.g. scholarly search), where it is often the only type of potential training data that is readily available in sufficient quantity. The intellectual merit of the project will lie in the development of principled machine learning methods that enable information systems to reliably learn from logs of the partial and biased feedback they produce. The theoretical basis for the research lies in deep connections to counterfactual and causal inference, exploiting the analogy between logs and controlled experiments with actions as treatments and the current system as the assignment mechanism. The research builds upon recent advances in counterfactual estimators, answering the question of how a new system would have performed, if it had been used instead of the system that logged the data. The project will develop new counterfactual estimators specifically designed for the action spaces typically encountered in information systems (e.g. rankings), new propensity models, and new counterfactual policy learning algorithms that incorporate both. Finally, to validate the real-world effectiveness of the research, the project will build the Localify system, which provides local music-event recommendations and personalized playlists. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.