ITR: Mining Text for General World Knowledge

University Of Rochester, Rochester NY

Investigators

Abstract

Despite significant advances in recent years in speech recognition generation technology and statistical language modeling, existing natural language systems are still limited to very specific, narrow domains, and totally lack common sense - the ability to "see the obvious" when interacting with a user. A major reason for this is the lack of a broad base of general world knowledge in current AI systems - knowledge such as that a sandwich is food (for. humans), while dinnerware is not; that dwellings usually have doors and walls; or, that when one person is killed by another, it is often with a gun; etc. This project will use previous work on mining linguistic knowledge from text as a springboard for tackling the problem of mining general world knowledge from texts. The methodology depends neither on "deep" text understanding nor on explicit occurrence of the desired general facts in the targeted corpora. Rather, the PI's approach elaborates on the idea that regularities observed in patterns of predication in texts generally reflect regularities in the world, particularly regularities in the way certain types of entities jointly participate in various events and relationships. While absolute statistical frequencies of such patterns can be severely misleading (people do not commit crimes, or have accidents or hold public office nearly as often as scanning of newspapers might suggest), the techniques that will be employed rely on conditional frequencies to obtain factually reliable hypotheses. The knowledge extracted will be cast in a formally interpretable propositional form, lending itself to certain and uncertain inference. This in turn will help "sanitize" the extracted knowledge, by revealing and helping to remedy apparent contradictions. Suitable corpora for this work include not only newspapers and other factual sources, but also realistic novels and writings for children - in fact, almost all electronically accessible texts are potentially useful, and no annotation will be required. While not all kinds of common-sense knowledge can be acquired in this way, the knowledge that can be acquired is very extensive, is essential to language understanding and common-sense reasoning, and is relatively close at hand. The kind of general knowledge to be mined from text corpora is not only useful, but essential in the long run for intelligent systems with some general linguistic competence and a modicum of common sense. Thus the work will bring a step closer the prospect of computers that genuinely understand their users.