Content Adaptive Encoding For Interactive Game Streaming

TL;DR

This paper introduces a novel content-adaptive resolution encoding method for interactive game streaming that uses CNN inference on past frame data to optimize resolution with minimal latency and computational overhead.

Contribution

It presents the first CAE approach for resolution adaptation in IGS using CNNs trained on encoding metadata, enabling low-latency, high-quality streaming.

Findings

Improves video quality by 2.3 Bjontegaard Delta-VMAF points over fixed resolution.

Operates with only 1ms CPU time per scene, ensuring no latency overhead.

Successfully deploys within HEVC encoding framework for practical use.

Abstract

Video-on-demand streaming has benefitted from \textit{content-adaptive encoding} (CAE), i.e., adaptation of resolution and/or quantization parameters for each scene based on convex hull optimization. However, CAE is very challenging to develop and deploy for interactive game streaming (IGS). Commercial IGS services impose ultra-low latency encoding with no lookahead or buffering, and have extremely tight compute constraints for any CAE algorithm execution. We propose the first CAE approach for resolution adaptation in IGS based on compact encoding metadata from past frames. Specifically, we train a convolutional neural network (CNN) to infer the best resolution from the options available for the upcoming scene based on a running window of aggregated coding block statistics from the current scene. By deploying the trained CNN within a practical IGS setup based on HEVC encoding, our proposal: (i) improves over the default fixed-resolution ladder of HEVC by 2.3 Bj{\o}ntegaard Delta-VMAF points; (ii) infers using 1ms of a single CPU core per scene, thereby having no latency overhead.