CRII: AF: Characterization and Complexity of Information Elicitation

University Of Colorado At Boulder, Boulder CO

Investigators

Abstract

The way one judges the accuracy of predictions can greatly impact what predictions people or computers make. For example, Glenn Brier argued in 1950 that the way meteorologists were evaluated would actually give them an incentive to distort the true probability of rain. Brier's study inspired a growing body of work in statistics, economics, and now computer science, which studies evaluation metrics that incentivize accurate reports from people or machines. These evaluation metrics are also used in machine learning, a branch of artificial intelligence, where a designer implicitly tells the computer what statistic to predict by providing only the evaluation metric itself. This project seeks to mathematically characterize this link between statistics and evaluation metrics, and moreover, to understand the computational and statistical difficulty of evaluating different statistics. A precise understanding of this link would provide new evaluation metrics with the potential to increase predictive power across a vast array of applications such as climate simulations and smart cities. In particular, metrics for statistics that quantify uncertainty or risk could improve decision making in many fields, including healthcare, engineering, and finance. A dominant algorithmic paradigm in machine learning, encompassing most regression techniques and classification algorithms, is that of empirical risk minimization (ERM): choosing a model from some class that best fits the data, according to some evaluation metric called a loss function. A thread of research in theoretical machine learning called property elicitation gives a mathematical formalism to describe the link between loss functions and their corresponding statistics. In these terms, this project seeks to characterize the statistics which have calibrated loss functions, and determine how many regression parameters or data points are required for the calibration to hold. These questions are particularly relevant to machine learning when restricting attention to certain classes of loss functions which can be easily optimized or which have desirable statistical learning guarantees. The class of statistics from mathematical finance known as risk measures, which are used to regulate banks, form an important focus of the project.