CAREER: Extended Reality Meets AI: Designing Interactions for Novel Human-AI Systems

University Of California-Santa Barbara, Santa Barbara CA

Investigators

Abstract

Research in human-computer interaction aims to develop user-centered systems that provide access to effective lifelong learning across the full range of knowledge domains and skills. A key area is learning motor or physical skills of tremendous practical (such as training for surgery), scientific (such as understanding motor memory) and cultural (such as music and sports) interest. To learn new motor skills or recover them after an injury, people often need intensive training or rehabilitation therapy with an expert who can instruct, assess performance, provide feedback, support, and motivate. Working with an expert can ensure learners achieve desired results, exercise correctly, avoid injury, and gain insights into their own performance. But an expert may not always be accessible or available, particularly in situations where demand for experts exceeds supply or when people have personal, monetary, or geographic limitations that prevent access. To deal with the absence of an expert, alternatives for learning at home include books, videos, apps, and online services (such as ClassPass, Peloton, Mirror). Although these options successfully provide explanations and demonstrations, none of them can provide meaningful real-time feedback to the trainees. This project leverages recent advances in Artificial Intelligence (AI) and eXtended Reality (XR) technologies to build new types of interactive motor skill training and rehabilitation systems where users receive real-time feedback from AI agents represented as 3D avatars. By building AI agents that emulate the real-time verbal and non-verbal behaviors of a human expert, the project will provide foundational knowledge on underlying principles that govern human-AI interactions in the context of motor skill learning (specifically, rehabilitation therapy). The project has the potential to broaden access to motor skill learning and rehabilitation at home by reducing financial or psychological barriers at low cost, with low intrusion, and at scale. Given that algorithms are increasingly making inroads into societally consequential domains, understanding how humans and AI agents interact has broad potential implications for the design of responsible and explainable AI. The goal of this project is to investigate the design of AI-XR (artificial intelligence - extended reality) systems for motor skill learning and rehab. Specifically, the project advances a new human-AI interface paradigm in which the AI is represented as a 3D humanoid agent in XR that can mimic the real-time verbal and non-verbal behaviors of a human trainer to support motor learning in the absence of a human expert. Human expert trainers make use of multimodal feedback through visual, auditory, and haptic channels when providing the necessary explanations, demonstrations, and corrections of physical motion performance. To emulate natural interactions between a trainee and an expert trainer, the AI agents will be designed to recognize and respond to a user’s physical actions in real-time to near-real-time, appear and act human-like, and use verbal and non-verbal behaviors (such as eye gaze, head nods, facial expressions, and gestures) to communicate. Developing AI agents for motor learning therefore presents numerous technical and generalizable research challenges, including (i) how to design effective multimodal instruction and feedback mechanisms and (ii) how to enable natural interactions between the user and an AI agent. To tackle these challenges, the project will implement new (i) algorithms and AI models for comprehensive body motion and error analysis; (ii) multimodal instruction and feedback techniques; and (iii) applications for fitness training and rehab that demonstrate the new human-AI interaction paradigm. The AI agent and system designs will be evaluated by performing controlled experiments with users. The evaluation of this research project will be through three applications for: (1) physical fitness, (2) sports injury rehabilitation, and (3) facial paralysis rehabilitation. Implementations of parts of this research in play- and peer-based educational activities for middle and high school students will be designed to be interactive and engaging, while eliciting problem solving and critical thinking. The goal is to guide the next generation to think deeply about AI and its implications on human behavior and by extension, on society. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.