EAGER: Converting Print Dictionaries to Machine-Interpretable Format

University Of Maryland, College Park, College Park MD

Investigators

Abstract

A dictionary documents the building blocks of a language -- its words and idiomatic phrases, with descriptions of their pronunciations, grammatical properties, meanings and uses -- and is an essential component of language documentation, together with a reference grammar and transcribed texts and recordings. Until recently, dictionaries were compiled and organized by hand, entered into some kind of typesetting system, and finally rendered in print form for use by scholars and language learners. Contemporary dictionaries are now compiled and organized electronically so that the information they contain can be used not only to produce stand-alone print artifacts, but also be integrated with the other components to ensure greater accuracy of the documentation as a whole, enable updates to be produced at regular intervals, and support the development of natural-language processing tools for the languages that are documented in this way. The goal of this exploratory project is to develop methods for machines to understand the implicit structure of the hundreds of extant print dictionaries of endangered and other low-resource languages as a critical first step in enabling their documentation to be of maximal usefulness to future generations. Print dictionaries use ordering, typeface and other formatting conventions to indicate the intended structure of dictionary entries. The first task of this project is to use optical character reading (OCR) software to convert those entries to machine-interpretable form so as to preserve the original formatting. The second is to develop software to convert the corrected OCR output into structured, machine-interpretable archive-standard formats. Because print formats vary widely across dictionaries, human intervention is required to inform the software about how to translate the implicit representations for a particular dictionary's entries into explicit ones. But such manual annotation is only required for a small part of the dictionary, as the formatting conventions are consistent across all of its entries, and once learned can be used to identify and correct errors and inconsistencies, and enable automated editing tasks like updating orthographies. The tool will be developed, tested and evaluated using print dictionaries of two indigenous languages of Latin America that were produced in the latter part of the twentieth century. This project is jointly supported by the Documenting Endangered Languages Program in the Behavioral and Cognitive Sciences Division and by the Robust Intelligence Program in the Information and Intelligent Systems Division.