CAREER: Mechanisms enabling the flexible expression of visual concepts

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Technologies for expressing ideas in visual form have been critically important throughout human history. From ancient cave paintings to modern digital graphics, such technologies lie at the heart of some of our most significant inventions, including art, writing, and mathematics. Despite the importance of such technologies, little is known about how the human mind is capable of using them in such varied ways. Perhaps the most basic and versatile of these technologies is drawing, which can be used to convey information about the visual world at many levels of abstraction, ranging from realistic illustrations to simplified diagrams. This project will harness large datasets and advanced data analysis techniques to develop a rigorous understanding of the mental processes involved when people use drawings to communicate visual concepts in different ways. Results from this project will advance our understanding of why people prefer to use certain kinds of images in some contexts and not others, with implications for how to design effective visualizations for a variety of applications, including STEM education and research. This project’s focus on the problem of how abstract ideas can be communicated in clear and accessible ways extends to its education plan. This plan encompasses two initiatives to develop inclusive learning experiences that promote computational literacy among K-12 and undergraduate students that have historically faced systemic obstacles to this training. Aim 1 of this project seeks to resolve a classic debate concerning whether drawings derive their meaning by resembling objects in the world (i.e., are image-like) or by being composed of discrete symbolic expressions (i.e., are language-like). The proposed experiments will evaluate the hypothesis that drawings are neither purely image-like nor purely language-like, but can vary strongly depending on what the illustrator can see, what they know, and what information they wish to communicate. To test this hypothesis, the proposed analyses will employ crowdsourcing and computer-vision techniques to measure the degree to which different drawings preserve perceptual information and/or are organized into discrete symbolic units. For example, some drawings may contain rich visual details that resist summarization in words, while other drawings may be entirely composed of simpler marks that can be readily described using words. Aim 2 will investigate the process by which people come up with new ways to visually communicate with each other over time. The proposed experiments will evaluate the hypothesis that drawings that initially resemble a concrete object tend to become increasingly abstract and symbol-like when people repeatedly communicate about it, reflecting shared goals and knowledge between communicators. The proposed analyses will measure consistency and variability in the resulting drawings, providing quantitative insight into the factors that affect the development of new symbolic systems for communication. Aim 3 seeks to better align the scientific training undergraduate psychology students receive to the reality of modern scientific practice in psychology. Specifically, it will integrate teaching of open-science best practices, exploratory data visualization, and model-based data analysis into the introductory statistics curriculum in Psychology at University of California San Diego, as well as collaborative final research projects to help students synthesize what they have learned and hone their communication skills. Aim 4 strives to broaden access to general education about artificial intelligence (AI) in historically underserved communities by partnering with local K-12 schools to develop learning experiences that illustrate the relevance of AI to students’ everyday lives, as well as how AI intersects with other disciplines, including the arts, psychology, medicine, and law. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.