Black-Box Science: Ideas and Insights for Learning-Based Statistical Inference

University Of Pittsburgh, Pittsburgh PA

Investigators

Abstract

This project seeks to develop essential tools that will allow scientists to better harness the full power of machine learning in practical scientific settings. In the current era of big data, machine learning algorithms have set themselves apart as excellent, accurate tools for modeling complex systems and predicting future outcomes. Determining why those particular algorithms actually work and how those predictions are generated has proven to be a much greater challenge, yet understanding these aspects is crucial for practical scientific use. For example, if an algorithm predicts that you are at risk for a particular disease, you will instinctively care less about the exact percentage chance you have of getting it and much more about why you are more likely to get it and whether there is something you could do to prevent getting it. This work will develop tools that allow scientists to more easily determine which variables most affect an algorithm's performance and whether some other collection of variables might offer an alternative but equally accurate explanation for the outcomes predicted. Various components of these algorithms will also be explored mathematically to determine whether some of them can be borrowed and inserted into simpler models in order to obtain predictions that are not only more accurate, but are also more easily explainable. This project seeks to develop efficient means of statistical inference within a machine learning context with an emphasis on random forests in particular. Specifically, a computationally efficient hypothesis test will be developed that allows for p-values for feature importance to be calculated with similar effort to the original algorithm. In addition to these tests, a framework for characterizing the uncertainty in the model selection process itself will be developed to provide insights into not just the optimal model obtained, but also to illustrate how many alternative models may exist with similar predictive power. Finally, an in-depth study on the fundamental role of randomness in supervised learning ensembles will be undertaken. Lessons learned about the helpful effects of such randomness will be utilized to boost performance of more traditional models in appropriate settings. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.