ERI: Generative Adversarial Networks for Video Coding

Santa Clara University, Santa Clara CA

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Video coding is an important technology that compresses video signals to save transmission bandwidth and to provide Internet users with visually pleasing decoded videos. Inspired by recent breakthroughs in deep learning, convolutional neural networks have been increasingly exploited into video coding algorithms to provide significant coding gains compared to conventional approaches. Nevertheless, existing convolutional neural network-based video coding schemes tend to generate blurry decoded images which are inconsistent with human perception, and the high computational complexity of these schemes hinders their deployment on power-constrained and computation resource-limited devices, such as smart phones and tablets. Recently, the generative adversarial network demonstrated its capability of decoding sharp and photo-realistic images at low bit rates, but little research has investigated its potential for video compression. This project will develop generative adversarial network-based video coding systems to enhance the coding efficiency, meanwhile providing decoded videos with high perceptual quality. The project will also investigate low-complexity algorithms to reduce the power consumption and to accelerate the inference speed of the proposed video coding systems so that they are suitable for mobile and low-latency applications. The success of the project is expected to accelerate the economic growth of streaming video services to benefit people’s daily professional and entertainment activities. It will advance surveillance video services to enhance public safety in places such as airport, offices, highway, and road intersections. The research activities of the project will provide opportunities to train graduate and undergraduate students through theses research, senior design projects, as well as machine learning and artificial intelligence courses. The research results of the project will be showcased in a summer engineering seminar program to motivate high school students to pursue science and engineering majors in college. This project will address two problems: (1) How to leverage temporal correlations among video frames and explore scene dynamics in a generative adversarial network-based video coding architecture? Two approaches are proposed: a hierarchical predictive coding approach, and a spatial-temporal coding architecture based on 3-dimensional convolution. Since most existing generative adversarial network models are for still image compression, the success of this research will open the door to generative adversarial network-based coding systems for video coding professionals. (2) How to reduce the computational complexity of deep video coding networks? Despite the performance benefits of deep learning-based video coding tools, few of them are currently being adopted in real-world scenarios. This is due to the high computational complexity, slow inference speed and the large graphic processing unit memory requirements associated with deep network computation. To address this problem, the proposed research will develop algorithms to reduce the complexity, model size and model parameters of deep learning-based video coding models via separable convolution operations. The research results will accelerate the deployment of deep video coding models in real-world applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.