JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model

TL;DR

JambaTalk is a novel hybrid Transformer-Mamba model for 3D talking head generation that improves lip-sync, facial expressions, and head movements, achieving state-of-the-art performance across multiple metrics.

Contribution

The paper introduces Jamba, a hybrid Transformer-Mamba architecture, combining SSM and Transformer strengths for comprehensive 3D talking head animation.

Findings

Achieves comparable or superior performance to state-of-the-art models.

Enhances motion variety and lip sync through multimodal integration.

Effectively handles long sequences with the Mamba architecture.

Abstract

In recent years, the talking head generation has become a focal point for researchers. Considerable effort is being made to refine lip-sync motion, capture expressive facial expressions, generate natural head poses, and achieve high-quality video. However, no single model has yet achieved equivalence across all quantitative and qualitative metrics. We introduce Jamba, a hybrid Transformer-Mamba model, to animate a 3D face. Mamba, a pioneering Structured State Space Model (SSM) architecture, was developed to overcome the limitations of conventional Transformer architectures, particularly in handling long sequences. This challenge has constrained traditional models. Jamba combines the advantages of both the Transformer and Mamba approaches, offering a comprehensive solution. Based on the foundational Jamba block, we present JambaTalk to enhance motion variety and lip sync through multimodal integration. Extensive experiments reveal that our method achieves performance comparable or superior to state-of-the-art models.