CHS: Medium: Collaborative Research: Augmented Reality Agents with Pervasive Awareness, Appearance, and Abilities

The University Of Central Florida Board Of Trustees, Orlando FL

Investigators

Abstract

Voice-based assistants that respond to commands of people can be thought of as virtual companions that are always standing by to play music, tell us the weather, turn the lights off, etc. While their powers to respond and act on our behalf are increasing, unlike real companions they are largely unaware of our presence, take little initiative, look like appliances rather than interaction partners, and have limited abilities to both respond to queries and to sense and control real objects around us. This project will develop Augmented Reality Agents (ARAs) to embody these voice-based assistants, making them more are aware of our appearance, emotions, and behaviors, giving dynamic visual representations that make us aware of their state and behaviors; and leveraging the growth in "Internet of Things" (IoT) infrastructure and devices to increase the breadth and depth of their awareness. Together, these advances will lead to more effective and accepted voice-based assistants, both in the home and beyond. Such ARAs have a number of potential applications, including healthcare (by increasing the realism of clinical simulation and training, or providing support for caregiving through remote communication and virtual companionship) and education (by being more engaging tutors or representing historically important individuals). To develop embodied Augmented Reality Agents (ARAs) with pervasive contextual awareness, appearance, and abilities the researchers will undertake a program of research aimed at the nexus of concepts and technologies associated with Augmented Reality (AR), Intelligent Virtual Agents (IVA), and the Internet of Things (IoT). To maximize the expected knowledge outcomes the researchers have organized their plans into three categories. First, they will develop new understanding of and priorities for ARA awareness, appearance, and abilities in a manner that does not require, nor depend on, a specific technological realization of automated behaviors. Second, they will use off-the-shelf and custom components to realize pervasive ARA functionality, to facilitate formative experiments related to basic ARA behaviors, and develop domain-specific applications and experiments. Third, the researchers will use application-specific realizations to assess the potential usefulness related to companionship and two areas of healthcare training: pediatric patient simulators and wide-area team-based medical training. The healthcare-focused work will leverage relevant courses at UF and UCF, and UCF's NSF REU center on the Internet of Things to engage students beyond their core team, in meaningful research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.