CHS: Small: Adaptive rendering and display for emerging immersive experiences

North Carolina State University, Raleigh NC

Investigators

Abstract

The fundamental architecture for rendering information on TVs and monitors has changed very little: the information is still sent as pixels, scanned row by row into a sequence of images called frames. At the same time, an increasingly important class of immersive applications is emerging: socially rich videoconferencing supports the nonverbal communication; in-situ learning has remote students using augmented reality to see lessons in context, such as biology in the garden; teleoperation allows doctors to deliver healthcare at distance. Unfortunately, traditional display architecture cannot deliver crucial features these applications require, including life-size, zero-delay remote windows; and multi-viewer, glasses-free experiences. This project investigates transformative, adaptive display architectures that might support these features. By designing an abstract “ideal display” using an API (Application Programming Interface), the research team will enable exploration of a diverse range of underlying display architectures for supporting emerging applications. In this process, it will convene workshops to build the multidisciplinary research relationships needed to meet these goals and begin addressing the dearth of women and minorities in computing with a special effort in the university's computer graphics class. To rekindle innovation in displays, the project will focus on five non-traditional, adaptive display techniques: frameless rendering, selective updates, local storage and processing, knowledge of the viewer, and high-level representations. Frameless rendering forms images with pixels representing different moments in time. This is particularly effective at reducing delay. Selective updates replace part of the image, rather than all of it, useful when images are too large for a full change. Local storage and processing allow older samples to be stored on the display, then combined with new samples just in time to make imagery with less delay and bandwidth. Higher level primitives like gradients and edges reduce bandwidth by succinctly describing local pixel distributions. Finally, knowledge of the viewer allows displays to render information only where and when it is needed, for example by using eye tracking. The project will apply these techniques toward three goals. First, to define an open API for the ideal perceptual display, which saturates all of the visual sense. This effort will drive future research by identifying underserved perceptual sensitivities, clarifying engineering tradeoffs in actual displays, and increasing the market for those displays. The second and third goals are to prototype two adaptive approximations of the ideal perceptual display on hybrid display-FPGA (Field Programmable Gate Arrays) systems, to test and inform our API. A frameless, foveated display will reduce detail in the periphery to reduce delay, particularly for large displays. A foveated light field display will vary color by view angle to support glasses-free stereo and multi-user viewing. The project will evaluate both prototypes with perceptual comparisons of them to more traditional displays. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.