III: Small: Distributed Reinforcement Learning over Complex Networks

Suny At Stony Brook, Stony Brook NY

Investigators

Abstract

In many distributed systems, a team of autonomous agents must collaborate in a complex environment, process massive amounts of streaming data, and simultaneously make optimal decisions. Traditional decision-making techniques can hardly tackle such a scenario, and reinforcement learning (RL) has been recently shown to be a promising decision-making technique for large-scale distributed systems. However, previous distributed RL models have failed to account for sensing and observing capabilities of agents, and thus rely on global information, which is not readily available in distributed environments. To fill this gap, this project aims to build a revolutionary, fully distributed RL system for large-scale networked systems without using global information. Toward this end, the project develops a novel theoretical framework, computational models, and scientific software tools needed to design, analyze, and test fully distributed RL algorithms. The algorithms will be further designed to be robust against dynamic environments and resilient to adversarial attacks, which will enable teams of multiple autonomous agents to reliably achieve their goals. The research will greatly impact real-world application areas where distributed machine learning algorithms and decision-making methods are needed. Typical examples include motion planning of teams of mobile robots, and coordination of networked smart devices in an IoT environment. The project promotes education and outreach activities, including broadening participation of female students in the field of machine learning, creating new courses, and designing research projects for K-12 students and undergraduates. The publications and software tools will be shared with the community to foster further research on distributed RL. The central goal of this project is to establish theoretical foundations for fully distributed RL algorithm design, analysis, and applications over large-scale networks. The key technical challenges include bridging the gap between the global and local observability settings and achieving resiliency in the presence of dynamic and untrustworthy communications. To achieve the technical objective and tackle technical challenges, the project investigates three main thrusts. The first thrust establishes the fundamental novel theory for the design of fully distributed RL by approximating global information via distributed estimation. The second thrust develops robust distributed RL algorithms against time-varying communication and sensing capabilities, communication delays, and asynchronous updating. The third thrust designs distributed RL algorithms that are resilient to adversaries and malicious attacks capable of introducing untrustworthy information into the communication network, by first designing communication-efficient RL algorithms in which each agent can transmit only low-dimensional states, and then designing resilient information fusion/aggregation approaches for small- and even single-dimensional cases. The project provides a suite of novel distributed RL algorithms which can be used in any applied area where fully distributed decision making and learning with streaming data and in adversarial environments are needed. Concurrently with the three main thrusts, the project also designs, develops, and maintains a software framework for empirically validating and studying distributed RL algorithms that the entire distributed RL community can use. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.