NSF-SNSF: VR-HRC: Virtual Reality-based Multi-Human-Multi-Robot Collaboration in Industrial Environments

Santa Clara University, Santa Clara CA

Investigators

Abstract

This joint National Science Foundation - Swiss National Science Foundation (NSF-SNSF) project aims to advance research in multi-human-multi-cobot collaboration in industrial settings. Collaborative Robots (Cobots) are increasingly being deployed in factories to assist workers with repetitive tasks or heavy lifting. Typically equipped with a Graphical User Interface and simulation software, these cobots are designed to be easily programmed by non-experts. However, while programming a single robot may be straightforward, coordinating multiple cobots to collaborate effectively with human workers can present challenges in a factory environment. During assembly processes, these robots often work alongside humans, helping to lift heavy components or provide necessary tools and materials as required. To effectively teach cobots their tasks, human operators must have a solid understanding of three-dimensional spatial processes. Simultaneously, the cobots must learn to interpret and adapt to human actions within their workspace. To address these challenges, this project introduces a Virtual Reality (VR) framework that facilitates collaboration between humans and cobots. The system aims to empower multiple users to interact using hand gestures, eye gaze, and speech within a physics-based VR simulation environment, thereby simplifying the process of teaching robots industrial tasks. The collaboration between Santa Clara University in the USA and the Dalle Molle Institute for Artificial Intelligence of the Scuola Universitaria Professionale della Svizzera Italiana in Switzerland will foster knowledge transfer and provide student researchers with cultural understanding, ultimately strengthening international research collaborations. This project will develop intent detection methods that leverage multimodal human demonstrations to generate actions for multiple collaborative robots (cobots) automatically. It involves creating an innovative physics-based virtual reality (VR) environment tailored for multi-cobot factory settings and employing generative AI techniques to model, generalize, and streamline the embodied reasoning between humans and multi-cobots. Additionally, the method incorporates human-in-the-loop approaches to refine the robot's behavior. The project encompasses three key thrusts: 1. Developing a VR interface for multi-human-multi-cobot collaboration, which will facilitate the collection of a multimodal multiperson dataset. 2. Establishing an intelligent framework that advances research in embodied reasoning and human-in-the-loop methodologies for the refinement of multi-cobot behavior. 3. Evaluating the proposed methods and systems through user studies, including a small pilot study with physical robots in real-world settings. This collaborative US-Swiss project is supported by the US National Science Foundation (NSF) and the Swiss National Science Foundation (SNSF), where NSF funds the US investigator and SNSF funds the partners in Switzerland. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.