CAREER: Integrating Perceptual and Linguistic Information in Models of Semantic Representation

Indiana University, Bloomington IN

Investigators

Abstract

Humans learn about the meanings of words and larger discourse units from repeated experience with both linguistic and perceptual information. However, current computational models of semantic learning and representation focus only on linguistic structure. This CAREER award explores how humans use multisensory perception and linguistic experience to organize semantic memory. Dr. Michael Jones at Indiana University will use specially designed web-based experiment protocols to collect large amounts of data about the perceptual structure of word referents. For example, a participant presented with a target word will be required to produce a list of verbal properties to describe the word (e.g., given DOG, a participant might produce "has four legs, has fur, barks" etc.). A second participant is then provided with only the list of properties and is required to guess what word is being described. Alternatively, the information to the second participant can be a drawing of what the target word represents to the first participant, who has configured predefined object components into a 2D or 3D display. By designing these experiments as internet based, Dr. Jones will collect very large amounts of data to design and test computer models of linguistic and perceptual integration during word learning. The models resulting from this research may provide a better understanding of the learning disorders characterized by language deficiencies (e.g. many forms of autism) and age-related disorders characterized by semantic disorganization (e.g., dementias such as Alzheimer's Disease). The integration of perceptual and linguistic information could also lead to better applied algorithms for information search (e.g., Internet search engines) if the computer representation can be made to approximate the semantic representation of the human doing the searching.