CAREER: Performance-Guided Synthesis of Virtual Environments for Personalized Training

George Mason University, Fairfax VA

Investigators

Abstract

Virtual Reality (VR) promises to provide a compelling, effective, and convenient means for training and reskilling the workforce. The objective of this project is to devise a computational design framework for guiding the synthesis of personalized virtual training environments for human performance. Designers and general users can intuitively apply this framework for synthesizing a variety of immersive and engaging VR training scenarios in a fully automatic, fast, scalable, and low-cost manner. As a showcase of the proposed framework, the project will demonstrate how to synthesize personalized virtual training environments for workforce training, such as safety inspection training for employees, workplace supervisors, and safety inspectors. This research project offers a novel interdisciplinary perspective for generating personalized VR training content by bringing together expertise and insights from human-computer interaction, virtual reality, computer graphics, machine learning, and optimization, as well as domain expertise in instructional design, safety, emergency management, and engineering. Designers and general users can use this framework to synthesize personalized VR training content for a variety of scenarios such as rehabilitation, safety training, disaster response training, as well as workforce training for different domains such as manufacturing, construction, logistics, transportation, retail management, and public safety. To achieve the project’s objectives, the researchers will address the following research question: (1) How to track human performance under virtual scenarios? A VR station will be set up which is capable of tracking multimodal body data such as gaze, body pose, hand movement, and the locomotion of a trainee in performing tasks in a virtual workplace. Based on such tracked data, machine learning models are trained to analyze and characterize the trainee’s skill levels. (2) How to synthesize virtual environments for personalized training? Optimization approaches will be formulated for synthesizing realistic and highly immersive virtual environments for adaptively training the trainee considering his/her skill levels, preferences, and personal training goals. (3) How to evaluate VR training effects? The investigator will evaluate the effectiveness of the synthesized VR training experiences by comparing with alternative training approaches quantitatively in terms of performance gain and knowledge retention; and qualitatively by obtaining domain experts’ and participants’ feedback about the effectiveness, enjoyment, and engagement of the training experience. To facilitate easy deployment and widespread adoption of personalized VR training, the investigator will publicly disseminate the software tools and toolkits devised through project websites, workshops, and research papers. In addition, the investigator will disseminate the research by (1) setting up an open-access VR station at the George Mason University’s Makerspace where faculty, students, and staff from various disciplines can conveniently experience personalized VR training; (2) organizing interdisciplinary VR training workshops and demo days to disseminate the VR training research findings, to showcase the VR training demos to the public, and to stimulate the research and adoption of VR training in different disciplines; (3) broadening the participation of first-generation underrepresented undergraduate students in computing and VR training research via a series of focused mentoring activities organized in collaboration with the Louis Stokes Alliance for Minority Participation program at the university. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.