Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation

TL;DR

This paper introduces a self-supervised approach using latent motion deviation and diffusion models to improve the realism and quality of co-speech gesture video generation, focusing on pixel-level motion details.

Contribution

It presents a novel self-supervised latent deviation method combined with diffusion models for more realistic gesture video synthesis, outperforming existing state-of-the-art techniques.

Findings

FGD, DIV, and FVD metrics improved from 2.7% to 4.5%

PSNR increased by 8.1%

SSIM improved by 2.5%

Abstract

Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods.