Self-Conditioned Probabilistic Learning of Video Rescaling

TL;DR

This paper introduces a self-conditioned probabilistic framework for joint video downscaling and upscaling that preserves more information and improves downstream task performance, extending to a lossy compression system.

Contribution

It proposes a novel framework that learns paired downscaling and upscaling processes simultaneously, enhancing information retention and task performance.

Findings

Outperforms existing methods in video rescaling and compression

Improves accuracy in downstream tasks like action recognition

Demonstrates effectiveness through extensive experiments

Abstract

Bicubic downscaling is a prevalent technique used to reduce the video storage burden or to accelerate the downstream processing speed. However, the inverse upscaling step is non-trivial, and the downscaled video may also deteriorate the performance of downstream tasks. In this paper, we propose a self-conditioned probabilistic framework for video rescaling to learn the paired downscaling and upscaling procedures simultaneously. During the training, we decrease the entropy of the information lost in the downscaling by maximizing its probability conditioned on the strong spatial-temporal prior information within the downscaled video. After optimization, the downscaled video by our framework preserves more meaningful information, which is beneficial for both the upscaling step and the downstream tasks, e.g., video action recognition task. We further extend the framework to a lossy video compression system, in which a gradient estimator for non-differential industrial lossy codecs is proposed for the end-to-end training of the whole system. Extensive experimental results demonstrate the superiority of our approach on video rescaling, video compression, and efficient action recognition tasks.