KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation

TL;DR

This paper introduces the KFusion of Dual-Domain model, which effectively generates facial landmarks from audio by separating emotional and contextual information, improving stability and efficiency in audio-driven talking face synthesis.

Contribution

The paper proposes a novel dual-domain fusion approach using the KAN model to enhance landmark generation from audio for talking face synthesis.

Findings

High efficiency compared to recent models

Effective separation of emotional and facial context information

Improved stability of landmark generation

Abstract

Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future.