CAREER: Scarlet: Learned Protocols and Functional Architectures for Low-Latency Internet Video

Stanford University, Stanford CA

Investigators

Abstract

This project will create new approaches to the design of systems that transmit video and audio over the Internet in real time, improving on the commercial state-of-the-art (e.g., Zoom, Google Meet, Skype, or FaceTime). The hypothesis underlying this project is that new techniques in academic computer science (such as purely functional architectures, machine learning, and low-latency real-time distributed systems design) are ready to be applied successfully to this area. The ultimate goal will be to create new systems that allow people to feel like they are in the same place, able to perform music or theater or have a robust conversation while separated over the Internet. The project plans to pursue scientific inquiries in three areas: protocols, codecs, and compression. For protocols, the core idea is to apply Model-Predictive Control and in-situ learning to study an individual network path and predict its packet-by-packet dynamics at a fine grain, in order to control near-term behavior. For codecs, the main hypothesis is that incorporating machine learning tightly in an encoder's inner rate-control algorithm can reduce end-to-end latency and produce a proxy objective function that is useful to the transport protocol. If these efforts are successful, this opens up an opportunity for compression: the idea that low latency can be used to substitute for bandwidth, and that real-time gaze information and gaze-contingent compression can lead to manyfold compression gains and "retina-quality" 360x180 (virtual reality) video streaming over realistic path capacities. The project will include in a series of live online theatrical and musical performances, using far-flung actors and audience members connected over the Internet. The performances will serve as a learning opportunity and forcing function for the research project, an opportunity for performers who can't travel to be physically collocated, and a public demonstration of the benefits of computer-science research in an everyday setting. The project will include new classes taught for undergraduates at Stanford University on the development of video and audio transmission technology. The project will maintain a source-code repository at https://github.com/stanford-stagecast. Code will be released publicly as it is written; data and results will be published in the academic literature and will be available publicly (linked from the repository) as well. The project website will be maintained for at least three years. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.