CAREER: Large-Scale Learning for Information Extraction

Ohio State University, The, Columbus OH

Investigators

Abstract

Much of human knowledge is encoded in text. This project aims to substantially advance the capability of machines to read large document collections and reason about the knowledge contained within them using minimal human effort. This will help people to overcome information overload and make better decisions by analyzing vital information that is locked away in unstructured text. Recent years have seen tremendous progress on tasks such as speech recognition and machine translation, by applying deep learning methods on massive, high-quality datasets; however, most available datasets for information extraction are either small or very noisy. The project will address these challenges by developing new methods that can learn more effectively from big, but noisy datasets that are constructed using distant supervision from an existing knowledge base (KB). To demonstrate the new methods' effectiveness, they will be used to support several novel applications. These include the detection of cyber-threats reported online and the analysis of experts' opinions about their severity. Recent studies have found that 75% of software vulnerabilities are first reported online, giving attackers time to exploit the vulnerability. Systems that can automatically read computer security blogs and analyze new threats could help security practitioners to track and prioritize them more effectively. The project includes a plan for integrating research and education. Outreach efforts aim to help attract a more diverse group of students to study computer science. These include hands-on workshops to expose freshmen to exciting natural language processing and artificial intelligence applications. The project will also help to engage advanced undergraduate students in research through new course materials on cutting-edge information extraction techniques. The research will address the machine reading data bottleneck by inventing new methods that can learn effectively from large, noisy datasets using distant supervision. These methods will address the challenge of label noise inherent in distant supervision by performing inference over latent variables during learning, filling in missing information, and resolving ambiguities. The approach combines the benefits of structured learning and neural networks; the structured learning component of the model can override noisy labels in cases where it is sufficiently confident -- this is balanced against a model of missing data in the KB. This will catalyze the rapid development of extractors for many new tasks and domains. To demonstrate this, extensive experiments will compare against state of the art methods using standard benchmark datasets for information extraction, including the Freebase/NYT corpus, TAC KBP datasets, and TACRED. Furthermore, the research will push the boundaries of minimal supervision for Information Extraction by exploring new applications that demonstrate the generality of the approach, including entity, relation and event extraction, time normalization and learning to extract a real-time feed of cyber-threat intelligence using distant supervision from the National Vulnerability Database (NVD). These applications are supported by a comprehensive evaluation plan that includes the development of new corpora and metrics. The project will produce a number of new datasets in addition to a toolkit for minimally supervised information extraction, that will be shared as open source software. This research effort will support the rapid development of information systems for a broad range of new tasks and domains using minimal human effort. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.