CAREER: Learning to Secure Cooperative Multi-Agent Learning Systems: Advanced Attacks and Robust Defenses

Tulane University, New Orleans LA

Investigators

Abstract

Cooperative multi-agent learning (MAL), where multiple intelligent agents learn to coordinate with each other and with humans, is emerging as a promising paradigm for solving some of the most challenging problems in various security and safety-critical domains, including transportation, power systems, robotics, and healthcare. The decentralized nature of MAL systems and agents' exploration behavior, however, introduce new vulnerabilities unseen in standalone machine learning systems and traditional distributed systems. This project aims to develop a data-driven approach to MAL security that can provide an adequate level of protection even in the presence of persistent, coordinated, and stealthy malicious insiders or external adversaries. The main novelty of the project is to go beyond heuristics-based attack and defense schemes by incorporating opponent modeling and adaptation into security-related decision-making in a principled way. The project contributes to the emerging fields of science of security and trustworthy artificial intelligence via a cross-disciplinary approach that integrates cybersecurity, multi-agent systems, machine learning, and cognitive science. The interdisciplinary nature of this project also brings unique opportunities for both curriculum development and student training. Developing robust defenses for large-scale MAL systems faces fundamental challenges induced by the hidden behavioral patterns of malicious agents, the dynamics and uncertainty of the environment, and the necessity of protecting benign agents' local data in many privacy-sensitive settings. This project tackles the challenges by incrementally developing a (machine) theory of mind for adversarial decision-making in three research thrusts. The first thrust develops learning-based targeted and untargeted attacks against federated and decentralized machine learning systems. These attacks first infer a world model from publicly available data and then apply model-based reinforcement learning to identify an adaptive attack policy that can fully exploit the vulnerabilities of the systems. The second thrust investigates a proactive defense framework that combines adversarial training and local adaptation, utilizing the automated attack framework developed in the first thrust as a simulator of adversaries to obtain robust defenses. The third thrust studies security in cooperative multi-agent reinforcement learning systems by addressing a set of new challenges, including complicated interactions among agents, non-stationarity, and partial observability. The goal is to understand how malicious attacks and deceptions can prevent benign agents from reaching a socially preferred outcome and how accounting for a higher order of beliefs can help an agent (benign or malicious) in both fully cooperative and mixed-motive settings. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.