RI: Small: Extending Verb Semantics with Causality towards Physical World

Michigan State University, East Lansing MI

Investigators

Abstract

With the emergence of a new generation of cognitive robots, the capability to communicate with these robots using natural language has become increasingly important. Verbal communication often involves the use of verbs, for example, to ask a robot to perform some tasks or to monitor some physical activities. Concrete action verbs often denote some change of state as a result of an action; for example, "slice a pizza" implies the state of the object pizza will be changed from one piece to several smaller pieces. The change of state can be perceived from the physical world through different sensors. Given a human utterance, if the robot can anticipate the potential change of the state signaled by the verbs, it can then actively sense the environment and better connect language with the perceived physical world such as who performs the action and what objects and locations are involved. This improved connection will benefit many applications relying on human-robot communication. Through a cognitive robot, this project will bring new educational experiences to K-12 students and encourage broader participation in engineering. This research project develops novel causality models for concrete action verbs to capture intended change of state of the physical world. It augments meanings of concrete verbs based on how they might change the environment (i.e., causality) and meanings of concrete nouns based on how they might be changed by actions (i.e., affordance). It incorporates causality models into learning and inference algorithms for grounding language to the physical world. This work will provide a new dimension to connect verb semantics to perception and action. Verb causality models will allow the robot to predict potential change of state from human linguistic utterances. This prediction will provide top-down information to guide visual processing and action modeling.