Rethinking Generative Human Video Coding with Implicit Motion Transformation

TL;DR

This paper introduces Implicit Motion Transformation (IMT) to improve generative human video coding, addressing the limitations of explicit motion guidance by transforming complex signals into implicit motion cues for better compression and reconstruction.

Contribution

It proposes a novel IMT paradigm that characterizes human body signals into compact features and transforms them into implicit motion guidance, enhancing GHVC performance.

Findings

IMT improves compression efficiency.

IMT achieves higher reconstruction fidelity.

Experimental results validate effectiveness.

Abstract

Beyond traditional hybrid-based video codec, generative video codec could achieve promising compression performance by evolving high-dimensional signals into compact feature representations for bitstream compactness at the encoder side and developing explicit motion fields as intermediate supervision for high-quality reconstruction at the decoder side. This paradigm has achieved significant success in face video compression. However, compared to facial videos, human body videos pose greater challenges due to their more complex and diverse motion patterns, i.e., when using explicit motion guidance for Generative Human Video Coding (GHVC), the reconstruction results could suffer severe distortions and inaccurate motion. As such, this paper highlights the limitations of explicit motion-based approaches for human body video compression and investigates the GHVC performance improvement with the aid of Implicit Motion Transformation, namely IMT. In particular, we propose to characterize complex human body signal into compact visual features and transform these features into implicit motion guidance for signal reconstruction. Experimental results demonstrate the effectiveness of the proposed IMT paradigm, which can facilitate GHVC to achieve high-efficiency compression and high-fidelity synthesis.