CompCog: A Machine Learning Approach to Human Perceptual Similarity

University Of Rochester, Rochester NY

Investigators

Abstract

Similarity is fundamental to nearly all aspects of human cognition. Perception uses similarity: when viewing a person's face, we (unconsciously) calculate its similarity to the faces of people we know in order to recognize who we are looking at. Categorization uses similarity: when judging whether a building was designed by the architect Frank Lloyd Wright, we calculate its similarity to buildings known to have been designed by Wright in order to make our best estimate. Reasoning and problem solving use similarity: when attempting to solve a calculus problem, we calculate its similarity to previous problems that we have encountered in order to determine a good solution strategy. However, how people calculate the similarity of two items is not yet understood. Which features of items do people use to calculate similarity? And how are the feature values of items compared in order to calculate similarity? This research project will use human experimentation and computational modeling to address these questions when items are viewed or grasped. A long-term benefit of the project is that a greater understanding of people's perceptual similarity judgments will provide a foundation for understanding how people calculate and use similarity in other areas of cognition. While conducting the research, undergraduate and graduate students will be mentored in the cross-disciplinary approach embodied in our investigation through participation in both experimental and computational aspects of the research project. This project focuses on developing a new empirical and theoretical foundation for understanding people's notions of similarity, particularly in the domain of perceptual similarity. The field of cognitive science is well aware that understanding similarity is essential to understanding human cognition. Despite this, the primary motivation for this project is the belief that, to date, cognitive science's approach to the study of similarity judgments is much too simple---the restricted class of similarity metrics considered by cognitive scientists is unlikely to scale to large, realistic settings. The primary hypothesis of this project is that the field of machine learning---especially the study of metric learning---can supply cognitive science with a rich array of complex and sophisticated models, models that will be necessary to accurately characterize people's similarity notions in large, realistic domains. Machine learning has pioneered the study of mathematically rigorous linear and nonlinear similarity metrics. We believe that the time is ripe for the field of cognitive science to make use of machine learning's recent advances. Machine learning's metric learning framework extends and elaborates the cognitive science approach in principled and innovative new directions. Indeed, this framework presents an unparalleled opportunity for cognitive science with the potential for transforming this field. Using the empirical and theoretical findings from machine learning, cognitive scientists can now begin to explore human notions of similarity in more complex and sophisticated ways---and in more realistic domains---than has ever been possible. We regard the research project as an early step for cognitive science towards a more sophisticated understanding of people's notions of similarity. Because the project cannot study similarity in all domains of human cognition, it concentrates on perception. Future work will need to develop further the models proposed and evaluated here. If successful, the program will establish an empirical and theoretical foundation that can subsequently be extended to many other areas of human cognition. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.