III:Medium:Computation and Communication Efficient Distributed Learning

Michigan State University, East Lansing MI

Investigators

Abstract

With the explosion of large-scale machine learning tasks and the increasing availability of computational resources, distributed learning has become the cornerstone for extracting information and knowledge from big data. Nodes in distributed learning need to communicate information. Thus, distributed learning faces challenges around computational efficiency similar to conventional machine learning. But it also faces the additional challenge of communication efficiency. These efficiency problems have greatly hindered the applications of distributed learning in large-scale machine learning tasks and complex computing environments, such as resource-limited edge computing. In this project, we embrace new challenges and opportunities to comprehensively study the computation and communication efficiency in distributed learning. The project’s novelties are providing new perspectives for developing and deploying efficient and scalable distributed learning algorithms in large-scale computing clusters with limited communication protocols and network bandwidth. Nowadays, as big data is ubiquitous, the project's impacts are to benefit many real-world applications from various disciplines such as computer science, social sciences and others areas. This project aims to tackle the major drawbacks in existing distributed learning algorithms from the efficiency perspective and greatly promote the efficiency and scalability of large-scale distributed learning. To achieve this goal, we systematically investigate two distributed learning paradigms, centralized and decentralized learning, as well as two major efficiency obstacles, computation and communication efficiency. To address these paradigms and obstacles, the project has three dedicated designed research directions. Each direction will dramatically extend the science through not only providing rigorous theoretical guarantees, but also comprehensive empirical studies in practical systems. The core intellectual is a comprehensive investigation on science and the design of novel methodologies to deepen our understanding on the efficiency, scalability, and practical usages of distributed learning systems and algorithms. The outcomes of this project will be: (1) New efficient and scalable distributed learning algorithms with state-of-the-art computation and communication efficiency, as well as predictive accuracy; (2) Theoretical analysis such as convergence rate and communication complexity; and (3) Open-source implementations of all key algorithms, systems, and frameworks. The proposed research will involve graduate and undergraduate students in pursuing their thesis or honor's projects. Discoveries and research findings of this project will be tightly integrated into several current and new courses. Instructional content will be created to enable fast distribution of our results to a wide audience, and tools will be built to help machine learning knowledge awareness and adoption. The findings of this project will be timely disseminated via multiple means such as a distributed learning repository, journal and conference publications, special purpose tutorials and workshops co-held at prominent conferences, and industrial participation such as internships. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.