Exploiting Communicative Signals to Summarize Information Graphics

University Of Delaware, Newark DE

Investigators

Abstract

In an effort to cope with information overload, much effort has been devoted in recent years to summarization, categorization, and retrieval of text documents. Yet relatively little attention has been paid to the graphics that appear in text documents, although they are an important resource and often convey information that is absent from any accompanying text. In this project the PI will develop a methodology for automatically summarizing informational (non-pictorial) graphics, such as bar charts and line graphs, that convey attributes of entities and relationships among the entities. Although some informational graphics are only intended to display data, the majority of them that appear in newspapers, magazines, and formal reports are intended to convey a message. This message captures the graphic's overall content and can serve as a summary of the graphic for storage, indexing, and retrieval. The underlying hypothesis of the project is that information graphics contain communicative signals that can be utilized to identify the message that the graphic conveys. These signals include the relative effort required for different perceptual tasks (since the easiest tasks are the ones a viewer will naturally perform and that will contribute the most to the message conveyed by the graphic), design choices such as graphic type, coloring, annotations, an exploded wedge in a pie chart, etc. (all of which may be used to draw attention to certain entities and relations in the graphic), and elements of the graphic's caption (although captions are of limited utility due to their generally ill-formed nature). The evidence about the graphic's message provided by the communicative signals will be entered into a Bayesian network that will attempt to deduce the graphic's message. As part of the research, a large corpus of information graphics will be collected and annotated with their identified messages, in order to compute the probability tables required by the Bayesian network. Evaluation experiments will be conducted to assess the system's ability to identify the message of an information graphic and the effectiveness of the message as a summary of the graphic. In addition to simple bar charts and line graphs, the project will investigate the summarization of complex graphics such as grouped charts and composite graphs consisting of several interrelated graphics. Although the project is only investigating the summarization of informational graphics, the PI expects the general approach and methodology developed here will provide insights into summarization methods for other kinds of graphics as well. Broader Impacts: The project will create technology that enables access to informational graphics in a digital library and thereby empowers individuals to better utilize the wealth of information available in graphical form. In addition, the outcomes of this research will include resources for use by others in the HCI community (e.g., the corpus of collected and annotated informational graphics will be made available to the research community). The research will also contribute to the training of future scientists, by providing interdisciplinary thesis topics that overlap computer science and cognitive science.