RI: Small: Linguistic Structure in Neural Sequence Models

Johns Hopkins University, Baltimore MD

Investigators

Abstract

Over the past 25 years, the field of artificial intelligence has made great strides in the ability to automatically analyze and generate sequential data. Much of this progress has come by building probabilistic models. For example, mathematical descriptions of how words are typically used in context are based on a scientific understanding of the relationships among letters, sounds, words, and phrases, thanks to the field of linguistics. Probabilistic models based on this understanding have allowed us to develop computational, data-driven methods for reasoning about the likely structure and meaning of sentences. In the same way, probabilistic models of sequences of events have led to computational methods for predicting the unfolding of future events and reconstructing the ordering of past ones. This project starts with sophisticated probabilistic models of linguistic structure and event sequences, and aims to make them more powerful, by using "deep learning" (neural networks) to increase their sensitivity to contextual effects. Deep learning has already recently had a revolutionary impact on artificial intelligence. This research will focus on using deep learning to enhance probabilistic models in settings where the model must discover structure that is not provided in its training data, such as the compositional units of language or the causal relations among events. The planned model design will not focus on hand-engineered features, but rather on broad representational choices. The overall architectures are motivated by certain basic notions that linguists and other modelers have found indispensable in their analyses of empirical data as follows: (1) stick-breaking processes that respect duality of patterning, the linguistic notion that a word's internal form is not necessarily related to its external usage but is governed by separate rules or by chance; (2) finite-state transducers that can capture local editing that transforms an input sequence into an output sequence; (3) context-free grammars that can model hierarchical structure to help explain word sequences; and (4) temporal point processes that can capture process intensity, where different events are competing to occur next, and combinations of earlier events combine to elevate or suppress the rates of later events. The project will infuse these probabilistic techniques with recurrent neural networks, in particular, long short-term memory (LSTM) networks. In some cases, exact inference in the resulting models will not be tractable, necessitating the design of Monte Carlo or variational approximations.