Accelerated distributed stochastic optimization methods and applications in machine learning

Rensselaer Polytechnic Institute, Troy NY

Investigators

Abstract

Machine learning, and in particular, deep learning, has become increasingly impactful in a wide range of applications, including face recognition, digital image classification, natural language processing, self-driving vehicles, and scientific computing. The success of deep learning largely depends on the availability of a huge amount of data. This "big" data, on one hand, enables successful learning of the underlying distributions of the data, and thus the learned model can yield high prediction accuracy on new data points that follow similar distributions. On the other hand, the huge amount of data raises great challenges when designing efficient numerical approaches. This project focuses on addressing the challenges that are caused by distributed "big" data that can contain private information, from the computational and mathematical perspectives. Research findings from this project will be included in graduate-level topics courses, undergraduate and graduate students will be trained in this field and will participate in this research, and a weekly seminar will be organized to exchange ideas relevant to this project. New computational approaches will be developed for training machine learning models on a cluster of computing nodes as well as solving decentralized multi-agent optimization problems that have the capacity to handle coupling constraints. The main goal is to design fast-convergent and communication-efficient optimization methods with theoretical guarantees for solving large-scale distributed machine learning problems. Accelerated compressed proximal stochastic gradient methods will be designed for distributed composite smooth stochastic problems, accelerated compressed stochastic subgradient methods will be designed for distributed nonconvex nonsmooth problems, optimal decentralized stochastic gradient methods will be designed for solving multi-agent optimization with nonlinear coupling constraints, and asynchronous implementations will be performed in the proposed methods in order to have high parallelization speed up. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.