RINGS: Bumblebee: A Neural Network Transformer Architecture for Summarization and Prediction in Interactive XR Applications

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Extended Reality (XR) applications tightly couple virtual content with the physical world through technologies such as virtual and augmented reality. These applications are compute-intensive, latency-sensitive, and bandwidth-hungry, making them ideal drivers for next-generation communication architectures. Current best-effort networking techniques struggle to meet the high demand of XR workloads leading to dropped or delayed network packets, observable jitter, and poor Quality-of-Experience. Fortunately, there are statistical correlations between past and future packets that go under-exploited in modern XR systems. The goal of this project is to develop a fast, resilient, and adaptive general neural network transformer-based architecture for interactive XR applications using machine learning (ML) to model, historically summarize, adapt to, predict, and then utilize streaming time-series data. The proposed modeling strategy is transformer-based, but it can represent context relationships going arbitrarily far back in time, something ordinarily infeasible, and it can also rapidly adapt to changing statistics via a novel adaptive final neural network layer. This work will result in a system that can transparently interpose network control messages to reduce bandwidth through forecasting and extrapolation. It will also help to mask packet delays and drops over wireless channels improving network coordination critical to many XR applications like AR guided surgery, search and rescue, digital telepresence and automotive heads up displays. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.