Convolutional Block Design for Learned Fractional Downsampling

TL;DR

This paper introduces a new convolutional block that enables learned fractional downsampling in CNNs, improving video compression efficiency by replacing traditional resizing methods with a differentiable, learnable approach.

Contribution

The authors propose a convolutional block with a differentiable resizer that allows CNNs to perform fractional resolution changes, enhancing video coding performance.

Findings

Improved PSNR, SSIM, and VMAF metrics over Lanczos resampling.

Enhanced coding efficiency in adaptive bitrate video streaming.

Flexible non-integer resizing within CNN architectures.

Abstract

The layers of convolutional neural networks (CNNs) can be used to alter the resolution of their inputs, but the scaling factors are limited to integer values. However, in many image and video processing applications, the ability to resize by a fractional factor would be advantageous. One example is conversion between resolutions standardized for video compression, such as from 1080p to 720p. To solve this problem, we propose an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. More concretely, the convolutional layer preserves the resolution of the input, while the resizing operation is fully handled by the resizer. In this way, any CNN architecture can be adapted for non-integer resizing. As an application, we replace the resizing convolutional layer of a modern deep downsampling model by the proposed building block, and apply it to an adaptive bitrate video streaming scenario. Our experimental results show that an improvement in coding efficiency over the conventional Lanczos algorithm is attained, in terms of PSNR, SSIM, and VMAF on test videos.