PersonaTalk: Bring Attention to Your Persona in Visual Dubbing
Longhao Zhang, Shuang Liang, Zhipeng Ge, Tianshu Hu

TL;DR
PersonaTalk is a novel attention-based framework for high-fidelity visual dubbing that emphasizes speaker's persona and style while ensuring accurate lip synchronization and facial detail preservation.
Contribution
It introduces a two-stage, style-aware, attention-driven approach for personalized visual dubbing, outperforming existing methods in quality and generality.
Findings
Outperforms state-of-the-art in visual quality and lip-sync accuracy.
Preserves facial details and speaker's persona effectively.
Achieves competitive results with person-specific methods.
Abstract
For audio-driven visual dubbing, it remains a considerable challenge to uphold and highlight speaker's persona while synthesizing accurate lip synchronization. Existing methods fall short of capturing speaker's unique speaking style or preserving facial details. In this paper, we present PersonaTalk, an attention-based two-stage framework, including geometry construction and face rendering, for high-fidelity and personalized visual dubbing. In the first stage, we propose a style-aware audio encoding module that injects speaking style into audio features through a cross-attention layer. The stylized audio features are then used to drive speaker's template geometry to obtain lip-synced geometries. In the second stage, a dual-attention face renderer is introduced to render textures for the target geometries. It consists of two parallel cross-attention layers, namely Lip-Attention and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsPersona Design and Applications · Human Pose and Action Recognition
MethodsSeventeen Ways to Call Uphold Helpline Full Guide USA 24 Hour Assistance
