PersonaTalk: Bring Attention to Your Persona in Visual Dubbing

TL;DR

PersonaTalk is a novel attention-based framework for high-fidelity visual dubbing that emphasizes speaker's persona and style while ensuring accurate lip synchronization and facial detail preservation.

Contribution

It introduces a two-stage, style-aware, attention-driven approach for personalized visual dubbing, outperforming existing methods in quality and generality.

Findings

Outperforms state-of-the-art in visual quality and lip-sync accuracy.

Preserves facial details and speaker's persona effectively.

Achieves competitive results with person-specific methods.

Abstract

For audio-driven visual dubbing, it remains a considerable challenge to uphold and highlight speaker's persona while synthesizing accurate lip synchronization. Existing methods fall short of capturing speaker's unique speaking style or preserving facial details. In this paper, we present PersonaTalk, an attention-based two-stage framework, including geometry construction and face rendering, for high-fidelity and personalized visual dubbing. In the first stage, we propose a style-aware audio encoding module that injects speaking style into audio features through a cross-attention layer. The stylized audio features are then used to drive speaker's template geometry to obtain lip-synced geometries. In the second stage, a dual-attention face renderer is introduced to render textures for the target geometries. It consists of two parallel cross-attention layers, namely Lip-Attention and Face-Attention, which respectively sample textures from different reference frames to render the entire face. With our innovative design, intricate facial details can be well preserved. Comprehensive experiments and user studies demonstrate our advantages over other state-of-the-art methods in terms of visual quality, lip-sync accuracy and persona preservation. Furthermore, as a person-generic framework, PersonaTalk can achieve competitive performance as state-of-the-art person-specific methods. Project Page: https://grisoon.github.io/PersonaTalk/.