ITR: Applying Translation Technology to Language Modeling

University Of Washington, Seattle WA

Investigators

Abstract

Virtually all systems that produce text, from speech recognition to natural language generation, use a language model as a core component in order to rank word strings by their well-formedness and appropriateness for a given context. These models are difficult to develop both because of algorithmic challenges specific to integration of multiple knowledge sources and the lack of robust language processing tools. The goal of this project is to develop models via new techniques for exploiting the information available in parallel multilingual corpora, i.e., translations of the same source in multiple languages. Such corpora implicitly encode a hidden, common core that can be uncovered using state-of-the-art estimation techniques. The project involves: i) automatic learning of structure within and across languages at multiple levels of abstraction: semantics, morphology, phonology, and paraphrasing, and ii) integration of the results into novel language model frameworks to address the problem of limited domain- and language-specific training data. The hypothesis is that, by sharing data and structure across languages and genres within a language, the resulting models will be richer and more robust. Such ideas were impossible to envision until recently; availability of multilingual corpora and increases in computing power make them now feasible. This project marries machine translation and speech recognition language modeling techniques, anticipating that the combination will lead to more powerful and general models. The research will facilitate rapid development of tools for less well studied languages and will immediately impact applications in mainstream languages ranging from information management to international collaboration to bilingual education. The results will also have implications for statistical modeling problems beyond language processing.