Versatile Recompression-Aware Perceptual Image Super-Resolution

TL;DR

This paper introduces VRPSR, a versatile perceptual super-resolution method that is aware of compression artifacts, using a diffusion-based codec simulator to improve image quality and reduce bitrate.

Contribution

It presents a novel framework combining perceptual SR with a diffusion model-based codec simulator, enabling joint optimization and compression-aware super-resolution.

Findings

Achieves 10-40% bitrate savings on real-world compression standards.

Utilizes a diffusion model to simulate versatile codecs for training.

Facilitates joint optimization of super-resolution and post-processing after recompression.

Abstract

Perceptual image super-resolution (SR) methods restore degraded images and produce sharp outputs. In practice, those outputs are usually recompressed for storage and transmission. Ignoring recompression is suboptimal as the downstream codec might add additional artifacts to restored images. However, jointly optimizing SR and recompression is challenging, as the codecs are not differentiable and vary in configuration. In this paper, we present \textbf{Versatile Recompression-Aware Perceptual Super-Resolution (VRPSR)}, which makes existing perceptual SR aware of versatile compression. First, we formulate compression as conditional text-to-image generation and utilize a pre-trained diffusion model to build a generalizable codec simulator. Next, we propose a set of training techniques tailored for perceptual SR, including optimizing the simulator using perceptual targets and adopting slightly compressed images as the training target. Empirically, our VRPSR achieves 10% - 40% bitrate savings based on Real-ESRGAN and S3Diff under H.264/H.265/H.266 single-picture (intra) compression. Besides, our VRPSR facilitates joint optimization of SR and the post-processing model after recompression.