III: Small: Multiple Device Collaborative Learning in Real Heterogeneous and Dynamic Environments

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

In collaborative learning, different devices such as smartphones or organizations like banks or hospitals learn together, using their own (sometimes private) data to build a shared model. This project tackles the challenge of scaling up this kind of learning for large, constantly changing, and diverse datasets. It proposes a move away from traditional computing systems, towards more flexible systems that can handle changing data types, decentralized computing, and different strategies between learning devices. Currently, problems arise from the diverse nature and volume of real-world data, like user photos, surveillance videos, or medical information. Present collaborative learning schemes often try to average out the learning updates from all devices, ignoring the individual characteristics of each device and their unique computing abilities. Moreover, these systems struggle with "free riding", where some participants benefit from the improved learning model without contributing any data. This difficulty calls for creating fitting incentives to encourage equal participation. The goal of this project is to improve collaborative learning by solving these issues, resulting in more efficient and fair learning systems that cater to individual devices' unique characteristics. This advancement goes beyond improving machine learning as it encourages data sharing, participation, and inclusivity, bringing about broader societal benefits. Current collaborative learning frameworks often achieve consensus by averaging model updates from participating agents, an approach that may disregard the unique attributes and diverse hardware capabilities of the agents involved. Such an oversight could lead to a mismatch between the model architecture and the hardware capabilities of specific devices, particularly edge devices with limited memory or computational power, thereby impeding efficient model training or underutilizing available resources. Additionally, the extant collaborative learning frameworks tend to overlook crucial factors such as algorithm trustworthiness and mechanism design. These challenges highlight the urgent need for a reimagined approach to collaborative learning. This project focuses on the development of a collaborative learning framework specifically designed for dynamic and diverse environments, with particular emphasis on standard hardware computing platforms, such as those comprising edge devices. The project's objectives are threefold: First, it seeks to innovate model-parallel collaborative learning by designing unique model architectures and efficient algorithms, underpinned by theoretical explorations employing a structured variational inference approach. Second, the project aims to facilitate the creation of practical, efficient training and communication learning algorithms for on-device usage. This aim will be achieved by introducing new algorithmic components and an authentic on-device testing platform. Lastly, the project intends to evaluate the system's trustworthiness and mechanism design in a decentralized setting. It will design incentives that promote data sharing and algorithm adoption, thereby maximizing benefits at both the community and individual levels. The project's targeted applications encompass disaster forecasting, AI-assisted clinical diagnosis, and treatment, and decentralized strategic decision-making. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.