RI: Small: A Cognitive Framework for Technical, Hard and Explainable Question Answering (THE-QA) with respect to Combined Textual and Visual Inputs

Arizona State University, Scottsdale AZ

Investigators

Abstract

Understanding of visual and textual inputs are important aspects of Artificial Intelligence systems. Often such inputs are presented together to instruct and explain. As examples, an intelligent robot might learn about its tasks and environment by observing both language and gesture; and an intelligent system addressing scientific questions must interpret figures and diagrams along with text. While there has been a lot of research concerning visual understanding and textual understanding in isolation, there has been very little research that addresses them jointly. This project is developing a framework for answering hard questions about combined visual and textual inputs, and providing supporting explanations. By developing a system that integrates visual and linguistic information for this task, the project could provide the basis for automated tutoring systems in K-12 education, and interpretable interfaces for the workers operating intelligent machines. The project will employ an integrated approach of deep model-based visual recognition and natural language processing, and knowledge representation and reasoning to develop a question answering engine and its components. It will create a challenge corpus that has visual and textual inputs and questions about those inputs given in natural language. It will provide a baseline for semantic image and text parsing and reasoning-based question answering systems. It will develop semantic parsing of non-continuous text items, such as figures, diagrams, and graphs. It will enhance semantic parsing to various formats of natural language text and questions. It will develop methods to acquire knowledge and reasoning with them for answering questions and providing explanations to the answers. Together these contributions of the project will advance Artificial General Intelligence and allow future service robots and personal mobile applications to understand combined visual and textual inputs. The findings from this project will advance the development of knowledge-driven, reasoning-based question answering by filling the current gap on how to efficiently conduct explainable probabilistic reasoning over deep models. This helps to overcome the fragility of the trained visual and textual understanding models. It will also uncover the intrinsic connections between deep model-based vision and language understanding algorithms and probabilistic knowledge representation and reasoning by exploring a joint solution for answering the hard questions. In general, this project may result in advances in multiple sub-fields of Artificial Intelligence; namely, computer vision, natural language processing, and question answering; and may impact others such as robotics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.