ITR: On-Demand Information Extraction

New York University, New York NY

Investigators

Abstract

The goal of Information Extraction (IE) is to extract from free text the essential relationships of a specific domain or scenario and record this information in tabular format. For example, in the management succession scenario, the essential information is the person, position, company name and date of management changes. The extraction process requires substantial scenario-dependent knowledge, and for most systems, this knowledge is currently created by hand. This laborious task, which sometimes takes months, is a major obstacle to the wider use of IE. The objective of this project is to fully automate the creation of scenario dependent knowledge. In the system under development, the user only specifies the scenario, and the system creates the knowledge needed to extract the information. This is being accomplished through the development of four components: 1) A broad-coverage name tagger, which identifies and classifies a wide variety of names and numeric expressions. It uses bootstrapping and active learning to overcome the sparseness problem. 2) Unsupervised learning of IE patterns and the associated word classes. 3) Automatic paraphrase discovery to find relationships between IE patterns. This uses comparable corpora and automatic identification of comparable phrases based on anchors such as names. 4) Discovery of rules for integrating information and building the table. This research will result in the creation of a new paradigm for IE. On-demand IE will provide useful and concise access to the information in text collections in a wide range of scenarios. We will make the knowledge and the systems available to the public after it has been shown to be successful. High-performance, rapidly portable extraction systems would have a major impact on the ways in which we can exploit textual knowledge, both everyday news and the scientific and technical literature. They would allow the creation of summaries, in table form, for collections of documents on a topic. Such tables could in turn serve as tools for highly-targeted document search or for mining the information for scientific research.