CRII: RI: Automatically Understanding the Messages and Goals of Visual Media

University Of Pittsburgh, Pittsburgh PA

Investigators

Abstract

This project develops technologies to interpret the visual rhetoric of images. The project advances computer vision through novel solutions to the novel problem of decoding the visual messages in advertisements and artistic photographs, and thus brings computer vision closer to its goal of being able to automatically understand visual content. From a practical standpoint, understanding visual rhetoric can be used to produce image descriptions for the visually impaired that align with how a human would label these images, and thus give them access to the rich content shown in newspapers or on TV. This project is tightly integrated with education. The work is interdisciplinary and can attract undergraduate students to the research from different fields. This research focuses on three media understanding tasks: (1) understanding the persuasive messages conveyed by artistic images and the strategies that those images use to convey their message; (2) exposing a photographer's bias towards their subject, e.g., determining whether a photograph portrays its subject in a positive or negative light; and (3) predicting what part of an artistic photograph a viewer might find most captivating or poignant. To enable decoding of artistic images, a large dataset is collected and annotated with a number of artistic properties and persuasion techniques that are intended for human understanding, then methods are developed to model visual symbolism in artistic images, as well as adapt positive/negative effect methods from sentiment analysis. To predict the photographer's bias towards a subject, a dataset of historical and modern portrayals of minorities and foreigners is collected, then an algorithm is created that reasons about body language and 3D layout and composition of the photo. To predict poignance, eyetracking data on a set of artistic images from famous photographers is collected, then semantic and connotation conflicts between the objects in the photographs are analyzed.