NRI: INT: Designing Effective Dialogue, Gaze, and Gesture Behaviors in a Social Robot that Supports Collaborative Learning in Middle School Mathematics

University Of Pittsburgh, Pittsburgh PA

Investigators

Abstract

When two students work together with a pedagogical agent, they tend to learn more as they talk to the agent and each other, explaining their reasoning and building on each other's ideas. What is not clear is whether and how the use of a physical robot rather than a virtual agent might improve the ways in which students interact and ultimately their learning. This award investigates how a robot's nonverbal behaviors might complement what it says to the students in order to prompt their thinking and develop their understanding. Using the robot to gaze at a student to encourage them to speak or make a mathematical gesture that helps the students clarify their own thinking might leverage the unique capabilities the robot brings to the interaction and ultimately enhance the effects of the robot's conversation with the students. This award investigates how the strategic design of two channels of communication of the robot – gesture and gaze – can be combined with dialogue to enhance middle school students' collaborative interactions within math. Ultimately, success in this project will contribute to broader understanding of how robots can be integrated effectively in learning environments in the future, as well as increase understanding of how co-robots can facilitate collaboration. Middle school students who participate in the studies will experience positive impact through the exposure to novel technologies and research, and transdisciplinary graduate and undergraduate students will be trained in the intersection of artificial intelligence, human-computer interaction, learning sciences, and cognitive psychology. This award brings together two theoretical frameworks, the ICAP theory of cognitive engagement and the Interactive Alignment Model of communication (IAM), to make two contributions: 1) How can data can be used such that the robot automatically learns effective social behaviors, and 2) What are empirically-tested desired robot behaviors that align to a particular framework? ICAP postulates that interactive activities where both students contribute constructively to the collaboration are best for learning, while IAM postulates that as collaborators pursue a successful communication, they will entrain to (or mimic) each other's dialogue choices and gestures. The award will use reinforcement learning to determine which specific gaze-dialogue behaviors (e.g., which human collaborator is being looked at during a particular type of dialogue) promote balanced interaction amongst the two human collaborators and when and how to introduce mathematically relevant terminology and gestures. This process is intended to create robot dialogue, gesture, and gaze patterns that are sensitive and responsive to individual differences in prior knowledge and motivation. Year 1 of the project will be spent developing the collaborative interaction with the robot. In Years 2 and 3 of the project, the adaptive gaze, gesture, and dialogue behaviors will be tested against non-adaptive control conditions. Finally, in Year 4, the project compares the optimal adaptive policies derived to a similar policy implemented within a virtual agent, exploring whether the embodied nature of the robot, combined with these strategically designed behaviors, offers significant advantages over a parallel agent. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.