CAREER: Understanding visual reasoning for visual communication

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

This project aims to advance the understanding of visual reasoning for visual communication. Visual reasoning enables people to translate visual input to abstract concepts. For example, to interpret which counties will receive more snowfall using a weather map, it is necessary to figure out which colors on the weather map indicate which amounts of snowfall. People have expectations about how visual features should map to concepts in visualizations, and it is harder for them to interpret visualizations that violate those expectations, even if mappings are clearly labeled. However, the nature of those expectations and their role in visual reasoning is not well-understood, so the design of information visualizations is often unprincipled and ad-hoc. With a better understanding of how visual reasoning works, it will be possible to design visualizations that fit its strengths and optimize visual communication. The investigators will address this problem by studying how people infer meaning from color in visualizations. This research can be translated to producing online tools for designing visualizations, which will improve STEM education and increase public literacy and engagement with science and technology. The education plan will use visual communication to make science more accessible and engaging through virtual reality (VR) and accompanying hands-on experiences with color and visualization, for both college undergraduates and middle-school students. The investigators will also support broadening participation of females in STEM through mentoring among the PI, graduate student, undergraduate intern, and middle school girls. The proposed research will address fundamental questions about how visual reasoning enables visual communication. In Objective 1, the investigators will study how people learn to associate perceptual features with novel concepts through environmental statistics. They will also examine how learned associations extend beyond perceptual input due to categorical structure in cognitive representations. In Objective 2, the team will study how to automatically estimate color–concept associations from image statistics. They will construct and evaluate new models that incorporate predictors based on visual input and cognitive representations of color categories. In the process of constructing these models, the team will develop new methods for quantifying graded category membership in a continuous space. By modeling human judgments, there is potential to develop new insights into how those judgments are made. In Objective 3, the team will investigate how people use color–concept associations to interpret visualizations, in a process called assignment inference. Assignment inference enables people to infer optimal mappings between colors and concepts in visualizations, but little is known about how this process works. It is proposed that it can be understood using accumulator models typically used in decision making research, which will forge new connections between the fields of decision making and information visualization. The results will increase knowledge of how people integrate information from multiple, conflicting sources and provide new knowledge about how to optimally design semantically interpretable visualizations. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.