JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1

TL;DR

This paper introduces JWB-DH-V1, a comprehensive benchmark dataset and evaluation protocol for joint whole-body talking avatar and speech generation, addressing current challenges in multi-modal consistency and region-specific performance assessment.

Contribution

It provides a large-scale multi-modal dataset and evaluation framework specifically designed for assessing joint audio-visual generation of whole-body avatars.

Findings

Performance disparities between face/hand-centric and whole-body generation.

Current models struggle with multi-modal consistency in whole-body synthesis.

Benchmark reveals key areas for future research in avatar generation.

Abstract

Recent advances in diffusion-based video generation have enabled photo-realistic short clips, but current methods still struggle to achieve multi-modal consistency when jointly generating whole-body motion and natural speech. Current approaches lack comprehensive evaluation frameworks that assess both visual and audio quality, and there are insufficient benchmarks for region-specific performance analysis. To address these gaps, we introduce the Joint Whole-Body Talking Avatar and Speech Generation Version I(JWB-DH-V1), comprising a large-scale multi-modal dataset with 10,000 unique identities across 2 million video samples, and an evaluation protocol for assessing joint audio-video generation of whole-body animatable avatars. Our evaluation of SOTA models reveals consistent performance disparities between face/hand-centric and whole-body performance, which incidates essential areas for future research. The dataset and evaluation tools are publicly available at https://github.com/deepreasonings/WholeBodyBenchmark.