Streaming of rendered content with adaptive frame rate and resolution

TL;DR

This paper presents an adaptive streaming system that dynamically adjusts frame rate and resolution using neural networks to optimize perceived quality and reduce rendering costs on bandwidth-limited devices.

Contribution

It introduces a neural network-based method for adaptively selecting frame rate and resolution in streaming rendered content, improving quality under bandwidth constraints.

Findings

Adaptive system improves perceptual quality over fixed-rate streaming.

Neural network accurately predicts optimal frame rate and resolution.

Method is codec-agnostic and minimally invasive to existing infrastructure.

Abstract

Streaming rendered content is an attractive way to bring high-quality graphics to billions of mobile devices that do not have sufficient rendering power. Existing solutions render content on a server at a fixed frame rate, typically 30 or 60 frames per second, and reduce resolution when bandwidth is restricted. However, this strategy leads to suboptimal rendering quality under the bandwidth constraints. In this work, we exploit the spatio-temporal limits of the human visual system to improve perceived quality while reducing rendering costs by adaptively adjusting both frame rate and resolution based on scene content and motion. Our approach is codec-agnostic and requires only minimal modifications to existing rendering infrastructure. We propose a system in which a lightweight neural network predicts the optimal combination of frame rate and resolution for a given transmission bandwidth, content, and motion velocity. This prediction significantly enhances perceptual quality while minimizing computational cost under bandwidth constraints. The network is trained on a large dataset of rendered content labeled with a perceptual video quality metric. The dataset and further information can be found at the project web page: https://www.cl.cam.ac.uk/research/rainbow/projects/adaptive_streaming/.