Towards Modality Transferable Visual Information Representation with Optimal Model Compression

TL;DR

This paper introduces a novel visual signal representation method using transferable modality and optimal model compression, enhancing video coding performance by transforming signals into a well-established modality.

Contribution

It proposes a new scheme for visual signal representation leveraging transferable modality and rate-utility optimization, integrated into versatile video coding standards.

Findings

Significantly improved representation capability demonstrated.

Effective model compression for transferable modality achieved.

Enhanced video coding performance observed in evaluations.

Abstract

Compactly representing the visual signals is of fundamental importance in various image/video-centered applications. Although numerous approaches were developed for improving the image and video coding performance by removing the redundancies within visual signals, much less work has been dedicated to the transformation of the visual signals to another well-established modality for better representation capability. In this paper, we propose a new scheme for visual signal representation that leverages the philosophy of transferable modality. In particular, the deep learning model, which characterizes and absorbs the statistics of the input scene with online training, could be efficiently represented in the sense of rate-utility optimization to serve as the enhancement layer in the bitstream. As such, the overall performance can be further guaranteed by optimizing the new modality incorporated. The proposed framework is implemented on the state-of-the-art video coding standard (i.e., versatile video coding), and significantly better representation capability has been observed based on extensive evaluations.