RI: Small: Modeling Multiple Modalities for Knowledge-Base Construction

University Of California-Irvine, Irvine CA

Investigators

Abstract

Information-rich documents are prevalent in many domains such as news articles, social media posts, online retail pages, healthcare records, financial reports, and scientific papers. Automatically extracting knowledge from such documents is useful in many applications, such as for answering questions, searching the web, automated dialogs, and analyzing trends. Existing machine learning methods focus only on the text in the documents, and ignore other information sources such as images, tables, and numbers. Thus, much of the information is not extracted, leading to incomplete knowledge and incorrect conclusions. This research advances research in machine learning and natural language processing to address these problems. With support for accurate extraction and reasoning, this project will pave the way for novel applications to domains with unstructured, multimodal documents. The specific aim of the project is to investigate a novel construction pipeline for knowledge bases, taking the first steps in combining textual and relational evidence with numerical, image, and tabular data. To address the many interconnected challenges therein, the project focuses on two sub-tasks. First, the team will extract new facts about an entity from a document, such as its attributes, by combining the different parts (text, images, and tables). Second, the team will develop models to identify missing relations in graphs that contain multimodal facts. For each task, the project includes plans to introduce new datasets, propose benchmark evaluations, and develop appropriate baselines. Further, the team will build upon recent advances in deep neural encoders to investigate machine learning approaches that learn unified, semantic embeddings to model multimodal data. With these contributions, the project will initiate a body of research in machine learning and natural language processing that uses unstructured multimodal data in all its forms for accurate knowledge extraction. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.