ITR: Visual Deictic Reference in a Collaborative Virtual Environment for Visual Search Training

Clemson University, Clemson SC

Investigators

Abstract

This research will study the interpersonal activity between trainer/trainee in a collaborative visual inspection Virtual Environment and will involve a new Collaborative Virtual Environment (CVE) capable of real-time capture and visualization of eye movements of one of the participants in the CVE. Of particular importance is the recording and real-time depiction of visual deictic references made with the eyes by either instructor or student in the environment. Deictic references are associated with pointing and verbal expressions (such as "look at this" or "see that?"). Collection of explicit visual deictic references, as made by the eyes, will facilitate task-based analysis and modeling of visual attention during human-human interaction in Virtual and Physical Reality. The main objective of this project is to develop a new interactive 3D visual inspection system in Virtual Reality, based on an existing single-person virtual aircraft inspection environment. To allow the measurement of visual deictic reference, it will extend the current single-person Virtual Environment (VE) to a Collaborative Virtual Environment (CVE) which will allow the simultaneous participation of two individuals. The main technological innovations are extensions to available eye tracking techniques to allow real-time capture and visualization of eye movements of one of the participants during simultaneous participation in the CVE. The fundamental exploration of visual deictic reference will be addressed during evaluation of visual inspection of the environment by the instructor (expert) and the trainee (novice). Existing approaches have failed to yield significant improvements in inspection performance. The successful completion of this research will provide a better understanding of the issues related to human-human interaction and training. Results from this research are expected to advance our understanding of complex multi-party environments where gaze-based communication is employed. The CVE will allow visualization and analysis of human attentional factors in shared-space environments. The study of visual deictic reference will lead to novel insights into human learning and behavior, notably learned improvement of visual search strategies. The research effort involves a multidisciplinary team of researchers with expertise in virtual reality systems, vision based research, human/machine systems design, inspection systems design, and training and computer supported collaborative work. Educational activities involving the curriculum and laboratory training will be closely integrated with the research, involving both undergraduate and graduate students. Although primarily focused on aircraft inspection, the anticipated results have the potential to impact a wide range of tasks in the maintenance and service industries, such as inspection of products in a manufacturing environment, flight crew systems training and training of air traffic control personnel, real-time military target acquisition, process control monitoring, and baggage inspection (aviation security).