RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content

TL;DR

RTSR is a low-complexity, CNN-based super-resolution model optimized for AV1 compressed videos, enabling real-time upscaling from 360p/540p to higher resolutions while maintaining high visual quality.

Contribution

The paper introduces RTSR, a novel low-complexity super-resolution method tailored for AV1 content, utilizing dual-teacher knowledge distillation for real-time performance.

Findings

Achieved the best trade-off between complexity and quality in AIM 2024 challenge.

Effective upscaling from 360p to 1080p and 540p to 4K.

Outperformed other submissions in PSNR, SSIM, and VMAF metrics.

Abstract

Super-resolution (SR) is a key technique for improving the visual quality of video content by increasing its spatial resolution while reconstructing fine details. SR has been employed in many applications including video streaming, where compressed low-resolution content is typically transmitted to end users and then reconstructed with a higher resolution and enhanced quality. To support real-time playback, it is important to implement fast SR models while preserving reconstruction quality; however most existing solutions, in particular those based on complex deep neural networks, fail to do so. To address this issue, this paper proposes a low-complexity SR method, RTSR, designed to enhance the visual quality of compressed video content, focusing on resolution up-scaling from a) 360p to 1080p and from b) 540p to 4K. The proposed approach utilizes a CNN-based network architecture, which was optimized for AV1 (SVT)-encoded content at various quantization levels based on a dual-teacher knowledge distillation method. This method was submitted to the AIM 2024 Video Super-Resolution Challenge, specifically targeting the Efficient/Mobile Real-Time Video Super-Resolution competition. It achieved the best trade-off between complexity and coding performance (measured in PSNR, SSIM and VMAF) among all six submissions. The code will be available soon.