RI: Small: A Data-Driven Framework to Sketch-to-Text Generation

University Of Washington, Seattle WA

Investigators

Abstract

The project aims to address the limitations of the current natural language generation (NLG) systems by seeking new data-driven approaches to modeling the contextual and creative dimensions of text composition. By taking a large collection of online text as an unstructured database of rhetorical patterns and linguistic creativity, the project develops a statistical generation engine that is capable of composing text with a new level of linguistic creativity and sophistication than what has been previously possible. Formulating sketch-to-text generation as a conceptual framework, the project investigates automatic composition of image captions and product descriptions as application scenarios. The project also explores new possibilities of human-computer collaborative writing, by developing an interactive search-based editor that will assist student writers to learn from a large collection of other people's writings. The technical outcome of the project has the potential to benefit our society in two ways: first, by advancing automatic image captioning for a wide variety of everyday photographs, it can contribute toward equal web access for visually impaired. Second, by enabling interactive search channels over a large-corpus of online writings, it can create new education experiences for training students' writing skills. The project is instrumental for supporting the PI's ongoing efforts in attracting and educating students from underrepresented groups. The proposed research is based on the premise that large-scale online writings, if used correctly, can be an enabling factor for sketch-to-text generation. The project consists of three fundamental research activities. First, the project proposes composition frames and elements as a new conceptual formalism to organize rhetorical patterns as building blocks, and develops unsupervised algorithms to extract them from a large-scale domain-specific corpus. Second, the project develops statistical approaches to differentiate literal language from figurative, with the specific goal of controlling the degree of literalness and creativity in the generated language. Finally, the proposed work designs scalable and robust inference algorithms for composition formulated as constrained optimization. Technical contributions include several unique resources to be shared with the research community, including a new large-scale corpus of image-caption pairs, and the database of learned composition frames and elements.