CAREER: End-User Robot Programming by Multimodal Instruction

Johns Hopkins University, Baltimore MD

Investigators

Abstract

This research will bring assistive robots within reach of average people by enabling such users to easily instruct them to help complete useful daily tasks, so that ultimately many lay people, especially older adults who wish to continue living independently, will be better able to benefit from the technology. It is projected that by 2060, the percent of the U.S. population aged 65 years or older will reach 23.5%. Consequently, the demand for healthcare resources and societal support are growing substantially. Our society, however, is presently experiencing a shortage of caregivers, and current caregivers are at increased risk for burnout. Project outcomes will allow users with little or no knowledge of coding or engineering to easily customize their robots through a variety of means (e.g., demonstrations) to perform tasks that meet their needs, such as organizing their shoes or cleaning the bathroom. Additional broader impact of the work will derive from educational activities that broaden participation in computing by mentoring students and dissemination of research findings to a wide age range of learners. To achieve these goals, this project will investigate the science of and fundamental technologies for robot programming by natural multimodal instruction. Human instructions are inherently multimodal, but the rich information embedded in them is not always adequately captured and utilized by robot systems. The research will computationally model multimodal instructional cues to characterize demonstration quality for improved learning, assist lay people in their task demonstrations, and enhance the user experience in the face of instructional difficulties. The work will create open-source systems for robot programming by multimodal demonstration, and produce quantitative and qualitative knowledge of how older adults naturally instruct robots and how they may use programmable robotic assistance to support them in their daily tasks at home. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.