Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony

TL;DR

This paper introduces Combo, a framework for co-speech 3D human motion generation that effectively handles multiple inputs and outputs, enabling customizable and harmonious facial and body movements with efficient adaptation.

Contribution

The paper presents a novel transformer-based design and a parameter-efficient fine-tuning method for holistic 3D human motion generation and adaptation.

Findings

High-quality motion generation demonstrated on BEAT2 and SHOW datasets.

Efficient transfer of identity and emotion in generated motions.

Effective coordination of facial expressions and body movements.

Abstract

In this paper, we propose a novel framework, Combo, for harmonious co-speech holistic 3D human motion generation and efficient customizable adaption. In particular, we identify that one fundamental challenge as the multiple-input-multiple-output (MIMO) nature of the generative model of interest. More concretely, on the input end, the model typically consumes both speech signals and character guidance (e.g., identity and emotion), which not only poses challenge on learning capacity but also hinders further adaptation to varying guidance; on the output end, holistic human motions mainly consist of facial expressions and body movements, which are inherently correlated but non-trivial to coordinate in current data-driven generation process. In response to the above challenge, we propose tailored designs to both ends. For the former, we propose to pre-train on data regarding a fixed identity with neutral emotion, and defer the incorporation of customizable conditions (identity and emotion) to fine-tuning stage, which is boosted by our novel X-Adapter for parameter-efficient fine-tuning. For the latter, we propose a simple yet effective transformer design, DU-Trans, which first divides into two branches to learn individual features of face expression and body movements, and then unites those to learn a joint bi-directional distribution and directly predicts combined coefficients. Evaluated on BEAT2 and SHOW datasets, Combo is highly effective in generating high-quality motions but also efficient in transferring identity and emotion. Project website: \href{https://xc-csc101.github.io/combo/}{Combo}.