RI: Small:Comp Cog: Broad-coverage semantic models of human sentence processing

Ohio State University, The, Columbus OH

Investigators

Abstract

Humans are a successful species in large part because they can pass knowledge about the world to one another using linguistic explanations. These explanations can be quite complex, involving nested generalizations about multiple classes of objects and events. Accurate models of how these relationships are decoded from natural language could further our understanding of how the brain works, and may allow non-programmer users to explain their desired products, goals and constraints to machines. Sentence processing experiments may provide an important window into the mechanisms of idea formation in language comprehension, but the human mind is extraordinarily sensitive to the strangeness of constructed stimuli used in experimentally controlled research designs, yielding potentially confounding effects arising from unexpected words or sentence structures. A common alternative is to use designs employing naturally-occurring stimuli with statistical controls, usually using one or more probabilistic measures of surprise during sentence processing. Unfortunately, existing probabilistic measures of surprise are based on overly simple models of sentence processing that are not connected to the nested structure of generalizations that a linguistic explanation may describe, and thus have severe limits as predictors of these kinds of frequency effects. This project will therefore develop a sentence processing model that decodes sentences into meanings using a human-like incremental probabilistic process. This model will then be used to control for frequency effects in neural activation, blood oxygenation and reading time data in order to isolate effects that can be attributed to the mechanical process of constructing and storing complex ideas during language comprehension. This project constructs a model of sentence processing that bases its processing decisions on mental representations of meanings rather than on words only. This means that the model will be less surprised by repeated nouns or pronouns when these words refer to a common entity which is prominent in a discourse. The project initially focuses on the development of a statistical sentence processing model which maintains several possible analyses of a sentence after each word is processed, each of which contains explicit representations of each discourse referent involved in a sentence meaning as a set of logical predicates adjacent to that referent in a graphical representation of the meaning. A subsequent version of the model compresses these context sets into vectors, which are passed through a recurrent neural network. The predictions of these models are compared against existing neural network language models used in natural language processing applications to ensure that their linguistic predictions are accurate. Incremental probabilities generated by these models are then used to estimate probabilistic surprise as a frequency control in predicting functional magnetic resonance (fMRI), electroencephalographic (EEG), eye-tracking, and reading-time observations in existing datasets, in order to isolate effects due to memory usage and other mechanistic factors. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.