AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation

TL;DR

AVI-Talking introduces a novel system that uses large language models and diffusion networks to generate expressive 3D talking faces aligned with speech, enhancing realism and emotional consistency.

Contribution

The paper presents a two-stage audio-visual instruction approach leveraging LLMs and diffusion models for expressive 3D talking face generation, improving interpretability and flexibility.

Findings

Effective synthesis of expressive facial movements

Enhanced emotional consistency in generated faces

Flexible instruction-based face customization

Abstract

While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker's speaking status remains challenging. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.