A Maximum Entropy-Based Approach to Discourse Parsing

University Of Southern California, Los Angeles CA

Investigators

Abstract

This is the first year funding of a three year continuing award. Despite a number of successful applications, current discourse parsers are still far from human performance levels. Two factors contribute to this situation. First, the relationship between discourse structure and lexicogrammar is insufficiently understood; we still do not know what lexicogrammatical features correlate with discourse structures and relations. Second, the discourse parsing algorithms implemented to date do not take advantage of formalism and algorithms specific to information and probability theories, which have been shown to produce impressive results in the field of syntactic parsing. The goal of this research is to address these two shortcomings. To this end, the PI will focus on the following two directions: (a) He will investigate empirically, using the principle of maximum entropy and the improved iterative scaling algorithm, the relationship between cue phrases, syntactic-, anaphoric-, and discourse-specific features/structures; (b) He will develop robust probability- and maximum-entropy-based discourse parsing algorithms. The empirical aspects of the project will contribute to a better understanding of the relationship between lexicogrammar, cohesion, and coherence, while the computational aspects will provide a thorough study of the features and search strategies that are best suited for probabilistic discourse parsing. This work will also provide computational linguists with robust discourse parsers, enabling them to investigate how the high-level structure of text can be exploited in a variety of natural language applications.