CAREER: Learning Optimization Algorithms from Data: Interpretability, Reliability, and Scalability

University Of Texas At Austin, Austin TX

Investigators

Abstract

Efficient and scalable optimization algorithms (a.k.a., optimizers) are the cornerstone of almost all computational fields. In many practical applications of optimization, one will repeatedly perform a certain type of optimization tasks over a specific distribution of data. Learning to optimize (L2O) is an emerging paradigm that automatically develops an optimization method (optimizer) by learning from its performance on a set of past optimization tasks. Then on solving new but similar optimization tasks, the learned optimizer can demonstrate many promising benefits including faster convergence and/or better solution quality. As a fast-growing new field, many open challenges remain concerning both L2O's theoretical underpinnings and its practical applicability. In particular, the learned optimizers are often hard to interpret, trust, and scale. The project targets those research gaps and expands to mid-term and long-term research directions pertaining to the foundations of L2O. Specifically, the project proposes a multi-pronged research agenda including: a novel symbolic representation that makes L2O lightweight and more interpretable; a Bayesian L2O modeling framework that can quantify optimizer uncertainty; new customized designs of L2O model architectures and regularizers that can robustly encode problem-specific priors; and a generic amalgamation scheme to bridge L2O training to classical optimizers as teachers. Each thrust addresses a unique aspect of L2O (representation, calibration, model design, and training strategy). Meanwhile, those thrusts are compatible with each other and can be applied together. The proposed efforts synergize cutting-edge technical advances from deep learning, symbolic learning, Bayesian optimization, and meta learning. Successful outcomes are expected to turn L2O into principled science as well as a mature tool for real applications. This project has an integrated plan of result dissemination, education, and outreach. In particular, all new algorithms resulting from the project will be integrated into the Open-L2O software package, developed and maintained by the PI's group. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.