CHS: Small: Minimal-Latency Tracking and Display for Head-Worn Augmented Reality Systems

University Of North Carolina At Chapel Hill, Chapel Hill NC

Investigators

Abstract

Augmented Reality (AR) enables computer images to be superimposed onto a user's view of his or her surroundings, most naturally via a head-worn display. For decades, many AR applications have been held back by bulky head-gear and inadequate displays. This situation is expected to change soon with the arrival of new commercially-available compact display designs that approach the form factor of eyeglasses. We expect these and other devices to spark renewed scientific and commercial interest in AR and its applications. AR systems, however, still suffer from a fatal flaw in that they are too slow for human vision. This slowness, or latency, hides in every subsystem, from a tracking-camera's capture, to deep rendering pipelines, to rendering frame buffers, to reformatting in the display controller. This latency causes causes misregistration between the synthetic imagery and its real-world counterparts. In other words, as you turn your head, the image that is supposed to remain superimposed on the real-world will start to move when it should not, and only later go back to where it should have stayed. This compromises the utility of the augmentation for many applications, in particular for high-precision uses such as aircraft maintenance or surgery. The proposed AR solution, combined with emerging comfortable, eyeglass-style head-worn displays should enable a wide range of applications to benefit from computer-generated visual augmentation: telepresence, medical examinations & procedures, maintenance, and navigation. Many applications that today use conventional displays for visualization will be able to use head-worn displays and reap the benefits of natural hand-eye coordination with augmented imagery anywhere the user looks. The project will be integrated into multiple courses at the University of North Carolina at Chapel Hill, which will stimulate student exploration of new directions in rendering, tracking, image acquisition and reconstruction, augmented reality and telepresence. Research products will expose to the broader research and development community a new approach to real-time 3D vision and 3D graphics - scanline stream processing, from camera capture to display update - which yields dramatically lower latencies and thus higher-fidelity alignment between real and augmented imagery. The project will exploit a characteristic of inexpensive cameras (continuous scanout, or "rolling shutter") that until now has been seen as a significant weakness of these devices, and will demonstrate how this is in fact a significant asset for providing more frequent updates of scene information. The project will replace the classic frame-by-frame processing of each subsystem with a unified scanline-based approach that significantly reduces latency. The user's tracked head pose will be updated at every scanline, just after each scanline is streamed in from a cluster of cameras affixed to the user's head-worn display. To match this tracking performance, the project will render the augmented imagery in scan line fragments by directly controlling the pixels in the fastest available display technology, that of Digital Micro-mirror Displays (DMD). All of these operations will be aimed to be eventually performed within a mobile device, communicating wirelessly with the user's eyeglass-stye display. Such mobile operation should empower precise augmentation over the user's visual field for a wide range of useful AR applications.