A Scalable and Accessible System for Automated Coaching of Human Motion

Dartmouth College, Hanover NH

Investigators

Abstract

The proposed work will develop a system for teaching physical motions such as sign language, dance, and human-robot collaboration automatically. Such motion skills support education and broad accessibility, can enhance well-being through creative expression and physical fitness, and enable new forms of work. Teaching motion often makes use of the intensive efforts of a human coach: the coach demonstrates a motion, evaluates the learner’s performance, and provides feedback and a tailored plan for practice to improve fluency. The proposed work aims to better understand how to develop systems that devise and carry out these components of the coaching process on their own. Such systems would make learning motion-based skills more accessible for those who do not have uninterrupted access to a dedicated human coach. The broader impact of the project also includes the training of the next generation of scientists at the intersection of psychology, cognitive science, education, and computer science. The goal of the work is to expand the capabilities of automatic teaching agents into the domain of physical motion, and to incorporate signals reflecting cognitive and emotional states of the learner into the system. Using pre-recorded motion examples from experts, and sensed actions and poses of the learner, this system will identify qualitative and quantitative differences between the teacher and learner. Low-level errors will be tracked to build an evolving cognitive model that measures the learner’s level of comfort with the process, as well as mastery of skills and combinations of skills. The project will develop computational tools that use this model to determine which feedback may be most effective to improve the learner’s performance. The project will also develop and evaluate hardware and software platforms that provide this feedback, with cues that may be presented in audio, visual, or tactile forms. To make the system as accessible as possible, the project will evaluate low-cost and ubiquitous approaches to sensing, including web cameras for sensing, and expert demonstrations parsed from internet videos. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.