Compressing Human Body Video with Interactive Semantics: A Generative Approach

TL;DR

This paper introduces a novel generative approach for compressing human body videos that allows interactive control over semantics, achieving high-quality reconstruction at ultra-low bitrates and enabling future metaverse applications.

Contribution

The paper presents a new encoder-decoder framework using a 3D human model for semantic-based compression and interactivity, outperforming existing standards at low bitrates.

Findings

Achieves promising compression at ultra-low bitrates compared to VVC and recent generative methods.

Enables interactive manipulation of human body videos without extra pre/post-processing.

Supports high-quality reconstruction through semantic-driven mesh evolution.

Abstract

In this paper, we propose to compress human body video with interactive semantics, which can facilitate video coding to be interactive and controllable by manipulating semantic-level representations embedded in the coded bitstream. In particular, the proposed encoder employs a 3D human model to disentangle nonlinear dynamics and complex motion of human body signal into a series of configurable embeddings, which are controllably edited, compactly compressed, and efficiently transmitted. Moreover, the proposed decoder can evolve the mesh-based motion fields from these decoded semantics to realize the high-quality human body video reconstruction. Experimental results illustrate that the proposed framework can achieve promising compression performance for human body videos at ultra-low bitrate ranges compared with the state-of-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes. Furthermore, the proposed framework enables interactive human body video coding without any additional pre-/post-manipulation processes, which is expected to shed light on metaverse-related digital human communication in the future.