Structure Induction for Manipulative and Interactive Tasks

Johns Hopkins University, Baltimore MD

Investigators

Abstract

Structure Induction for Manipulative and Interactive Tasks This project will develop methods for inferring models, similar to those traditionally used for speech and language, for structured, repetitive, interactive manipulation tasks. Such tasks are typically composed of more elementary tasks that follow a nominal plan of action to achieve a set of desired outcomes. Examples include assembly tasks, various forms of surgery, gesture-based interfaces, maneuvering in traffic, sports, dancing or gymnastics to name a few. The project will be carried out in the context of observing and modeling minimally invasive surgical procedures. Motion and video streams will be acquired from surgeons during simulated surgical tasks using the da Vinci surgical robot. Surgical tasks will be represented using grammars on a set of simple, elementary surgical gestures modeled using finite-state machines. Both the elementary gestures and the task-dependent grammatical constraints will be inferred directly from data. A key element of the proposed approach is to first develop accurate models from simple, low-dimensional datasets such as direct measurements of surgical motion, and then to use these models to boostrap model training to more complex, high-dimensional synchronized video of the same procedure. The core research problems to be addressed are thus: Motion sequence modeling for elementary tasks: Develop algorithms that are able to learn finite-state machines modeling simple, context-independent elementary motion or manipulation tasks in terms of atomic gestures using a limited set of manually labeled motion data. Multi-modal models of elementary tasks: Create effective methods for augmenting elementary task models with complementary and contextual features from synchronized video data. Grammar induction: Use elementary task models to automatically label more, larger data sequences and infer high-level grammatical structures from these labelings. Model bootstrapping: Use elementary task and grammatical models to label even larger data corpi that are then used to retrain elemetary task models using only video data. In summary, this project will study the problem of modeling structured human manipulative actions such as those that occur during minimally invasive surgery. Using statistical modeling methods originally developed for speech understanding, models for the language of surgical motion will be created and evaluated. These models will first be created using motion measurements from a surgical robot, but will then be transferred to video data from the same system. These specific models will have applicability to surgical training, skill assessment, and robotic surgical augmentation. More generally, this project will advance our ability to develop structured models for understanding com-plex data sources, which has wide ranging implications. It will facilitate a variety of applications that require understanding of human action such gesture-based systems for human-machine interaction, methods for interpreting or detecting action in video surveillance data, or assessment and measurement of movement for bio-mechanics and medicine. At the same time, it will advance the basic science of modeling large and complex data streams such as video, text, speech, or medical images. URL: http://www.cs.jhu.edu/CIRL/projects/SurgicalModeling/