CAREER: Resolving Lexical Ambiguities in Natural Language Processing

Johns Hopkins University, Baltimore MD

Investigators

Abstract

This is the first year of funding of a 4 year, continuing award. One of the major roadblocks in the efficient and accurate communication between humans and machines is the resolution of the ambiguity inherent in natural languages. A major bottleneck in developing solutions is the severe shortage of training data that distinguishes word senses, and the high cost of inputting this information manually. A focus of this project is the development of unsupervised and minimally supervised algorithms for acquiring such skills without costly hand-tagged training data. Such methods will exploit the distribution properties observed in very large text corpora (over 10 billion words); the PI will also investigate richer representations of feature space, class models, smoothing methods and learning algorithms specialized for classification in very high-dimensional featurespaces. In addition to the problem of word-sense disambiguation, this project will explore shared solutions to a closely related set of lexical-ambiguity tasks including spelling correction, propername classification, capitalization restoration, accent and diacritic restoration for, diverse languages, vowel restoration in Hebrew and Arabic, speech synthesis on homographs, lexical choice in machine translation, and certain aspects of choosing among phonetically confusable candidates in speech recognition. These diverse problems are not normally recognized as being members of the same class, and this project seeks to exploit the synergies present by developing methods and training data on one member of the class and utilizing the methods and data on other. problems in the class. Thus this unified approach offers the potential for rapid parallel progress on key problems in human-computer interaction and information extraction.