RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

This project develops new technologies at the interface of computer vision and natural language processing to understand text-to-image relationships. For example, given a captioned image, the project develops techniques which determine which words (e.g. "woman talking on phone", "The farther vehicle") correspond to which image parts. From robotics to human-computer interaction, there are numerous real-world tasks that benefit from practical systems to identify objects in scenes based on language and understand language based on visual context. In particular, the project develops the first language-based image authoring tool which allows users to edit or synthesize realistic imagery using only natural language (e.g. "delete the garbage truck from this photo" or "make an image with three boys chasing a shaggy dog"). Beyond the immediate impact of creating new ways for users to access and author digital images, the broader impacts of this work include three focus areas: the development of new benchmarks for the vision and language communities, outreach and undergraduate research, and leadership in promoting diversity. At the core of the project are new techniques for large-scale text-to-image reference resolution (TIRR) that enable systems to automatically identify the image regions that depict entities described in natural language sentences or commands. These techniques advance image interpretation by enabling systems to perform partial matching between images and sentences, referring expression understanding, and image-based question answering. They also advance image manipulation by enabling systems that can synthesize images starting from a textual description, or modify images based on natural language commands. The main technical contributions of the project are: (1) benchmark datasets for TIRR with comprehensive large-scale gold standard annotations that will make TIRR a standard task for recognition; (2) principled new representations for text-to-image annotations that expose the compositional nature of language using the formalism of the denotation graph; (3) new models for TIRR that perform an explicit alignment (grounding) of words and phrases to image regions guided by the structure of the denotation graph; (4) applications of TIRR methods to referring expression understanding and visual question answering; and (5) applications of TIRR to image creation and manipulation based on natural language input.