CAREER: Strengthening the Theoretical Foundations of Federated Learning: Utilizing Underlying Data Statistics in Mitigating Heterogeneity and Client Faults

Northeastern University, Boston MA

Investigators

Abstract

Real-world applications that would benefit from improved machine learning encompass a wide range of industries and domains such as healthcare, autonomous vehicles, natural language processing, and manufacturing and industry. Distributed machine learning has gained significant momentum in recent years due to the increasing need for real-time data processing, low latency, and privacy concerns. The rapid development of edge devices broadens the applicability of distributed machine learning yet brings nontrivial challenges that call for revisiting the fundamental principles and algorithm designs for federated learning. The research goal of this project is to consolidate the theoretical foundations and to enrich the algorithmic toolbox of distributed machine learning with a focus on enhancing its resilience against a wide range of data heterogeneity, system imperfection (or faults), and external attacks. The educational objective of this project is to promote the importance of principled mathematical thinking for solving real-world problems in machine learning among the next generation of machine learning practitioners and researchers. Federated learning is a rapidly evolving distributed machine learning approach that facilitates global model training without the necessity of sharing raw local data. Most existing theoretical analysis of federated learning is derived from an optimization perspective but the underlying statistical structure of the dataset is mostly overlooked. This often leads to misalignment between the pessimistic theoretical predictions and empirical success. In addition, recent work suggests that the bounded gradient dissimilarity conditions, which are frequently adopted in federated learning analysis, may be too pessimistic for practical applications. Motivated by our preliminary successes and backed by extensive prior work, this proposal aims to strengthen the theoretical foundations of federated learning and to enhance its resilience against a wide range of data heterogeneity and system failures, by leveraging the underlying structures of the federated datasets and by designing new algorithms. Towards this goal we will employ and innovate tools from statistical learning, distributed computing, high-dimensional probabilities, and optimization. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.