RI:Small: Neuro-PGM: Neuro-Symbolic Integration with Probabilistic Graphical Models Towards Data Efficient, Generalizable, and Interpretable Deep Learning

Rensselaer Polytechnic Institute, Troy NY

Investigators

Abstract

Despite the impressive performance of modern deep learning models across various domains, they still suffer from several fundamental limitations, including high data dependency, poor generalization, and low interpretability. These limitations primarily arise from the data-driven nature of deep models and their inability to effectively leverage prior knowledge. Traditional symbolic AI models successfully incorporate different types of prior knowledge, offer interpretability, and exhibit good generalization. However, they suffer from the slow and difficult process of extracting and structuring knowledge and, therefore, do not scale up as well. To address these deficiencies and increase the applicability of deep learning models to many real-world scenarios, this project will create a hybrid AI model that systematically integrates modern deep learning with probabilistic graphical models (PGMs; a way of encoding relationships between variables to transit knowledge). Thus, the prior knowledge encoded in the PGMs works synergistically with the data encoded in the deep learning model. This approach, effectively incorporates prior knowledge into deep learning models and could greatly expand their utility to a wide range of data-scarce yet knowledge-rich applications—such as those in manufacturing, scientific discovery, medicine, and defense. However, current efforts to address these limitations tend to be heuristic, ad hoc, and narrow in scope, both in the types of domain knowledge considered and in the methods used for knowledge integration. This project proposes a systematic and unified approach to identifying, encoding, and integrating prior knowledge with data. Specifically, the project systematically categorizes and organizes diverse forms of prior knowledge—including theoretical and experiential knowledge—drawn from a broad range of sources. It employs PGMs as a unified framework to represent various types of knowledge (e.g., mathematical equations, logical rules, and knowledge graphs) and develops learning algorithms to automatically encode this knowledge into the PGM. Finally, the project introduces complementary methods for integrating PGM-encoded knowledge with data within deep learning models at multiple levels, including decision level, architectural level, and training level. The proposed framework is evaluated on both vision-based tasks, such as human nonverbal behavior analysis and recognition, and non-vision tasks in domains like scientific discovery and manufacturing, with a focus on improving data efficiency, generalization, or interpretability. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.