CAREER: Combining Learning and Reasoning for Spatial Language Understanding

Tulane University, New Orleans LA

Investigators

Abstract

The main goal of this project is to teach machines to understand human language when it contains spatial information. For example, to understand the statement, "Give me the book on AI on the table to your left," a robot needs to understand that the first "on" expresses the topic of the book, while the second "on" and the rest of the sentence convey spatial information. Next, it needs to work out the visual meaning of "on" to identify the correct book. Spatial language understanding is challenging because it requires meaning disambiguation, recognizing the links between referenced objects, and in most cases, background knowledge and common sense. This is particularly true for more complex sentences that include nesting relations or idioms containing spatial terms, like "she is the top expert in her field." The goal of this research is to develop techniques for the integration of machine learning and reasoning to advance spatial language comprehension. Solving this problem will provide tangible benefits to society in a wide variety of applications including healthcare (e.g., extracting biomedical information from patient reports and images), navigation and communication with assistant robots (particularly in risky situations like firefighter robots), situational awareness, information retrieval systems using various types of data, and geographical information systems. Providing reasoning capabilities to machine learning models is highly challenging. The proposed research advances this issue in an important and broad sub-problem: spatial language understanding. To this aim, a generic domain-independent symbolic spatial meaning representation will be introduced, which covers the main spatial semantic concepts for a variety of tasks. Deep structured and relational learning techniques will be developed to obtain such representations. The proposed techniques will facilitate indirect supervision by exploiting visual resources, will use ontologies that convey common sense and will integrate the axioms of spatial qualitative reasoning in training structured representations. Such an approach helps to avoid massive and complex data annotation for training structured and complex models. This research will evaluate whether using formal spatial calculi models as an intermediate structured representation will improve spatial reasoning and deeper learning. The interaction between learning and spatial reasoning will be investigated, while exploiting both symbolic and sub-symbolic representations. The impact of integrating explicit spatial reasoning will be tested on textual and visual question-answering tasks with a focus on locative questions. The outcome of this research will advance the generalizability of current state-of-the-art deep learning models at a time when there is a lack of training examples for complex unobserved situations. This work will help bridge the gap between symbolic AI and deep ML to advance spatial language understanding. The project will support research and education for graduate and undergraduate students and increase the contribution of women and minorities in STEM. This award is jointly funded by the Division of Information and Intelligent Systems in the Directorate for Computer & Information Science & Engineering and the Established Program to Stimulate Competitive Research in the Office of Integrative Activities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.