An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal

TL;DR

This paper introduces VCM, a scalable video coding paradigm that combines predictive and generative models to efficiently compress video signals for both human and machine vision, improving reconstruction quality and recognition accuracy.

Contribution

The paper proposes a novel scalable coding approach using deep generative models guided by learned motion patterns, bridging feature and signal compression for dual vision tasks.

Findings

Better reconstruction quality than traditional codecs (0.0063 SSIM gain)

Enhanced action recognition accuracy (9.4% gain) on compressed videos

Effective use of sparse motion patterns for high-level vision tasks

Abstract

In this paper, we study a new problem arising from the emerging MPEG standardization effort Video Coding for Machine (VCM), which aims to bridge the gap between visual feature compression and classical video coding. VCM is committed to address the requirement of compact signal representation for both machine and human vision in a more or less scalable way. To this end, we make endeavors in leveraging the strength of predictive and generative models to support advanced compression techniques for both machine and human vision tasks simultaneously, in which visual features serve as a bridge to connect signal-level and task-level compact representations in a scalable manner. Specifically, we employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames via a generative model, relying on the appearance of the coded key frames. Meanwhile, the sparse motion pattern is compact and highly effective for high-level vision tasks, e.g. action recognition. Experimental results demonstrate that our method yields much better reconstruction quality compared with the traditional video codecs (0.0063 gain in SSIM), as well as state-of-the-art action recognition performance over highly compressed videos (9.4% gain in recognition accuracy), which showcases a promising paradigm of coding signal for both human and machine vision.