Instruct-NeuralTalker: Editing Audio-Driven Talking Radiance Fields with Instructions

TL;DR

This paper introduces Instruct-NeuralTalker, a real-time, interactive framework for editing audio-driven talking radiance fields using human instructions, combining diffusion models and optimization to enhance personalization and quality.

Contribution

It presents a novel interactive editing method for neural talking face models that incorporates human instructions and real-time rendering capabilities.

Findings

Achieves up to 30FPS rendering on consumer hardware.

Significantly improves rendering quality over existing methods.

Enables controllable, personalized talking face editing.

Abstract

Recent neural talking radiance field methods have shown great success in photorealistic audio-driven talking face synthesis. In this paper, we propose a novel interactive framework that utilizes human instructions to edit such implicit neural representations to achieve real-time personalized talking face generation. Given a short speech video, we first build an efficient talking radiance field, and then apply the latest conditional diffusion model for image editing based on the given instructions and guiding implicit representation optimization towards the editing target. To ensure audio-lip synchronization during the editing process, we propose an iterative dataset updating strategy and utilize a lip-edge loss to constrain changes in the lip region. We also introduce a lightweight refinement network for complementing image details and achieving controllable detail generation in the final rendered image. Our method also enables real-time rendering at up to 30FPS on consumer hardware. Multiple metrics and user verification show that our approach provides a significant improvement in rendering quality compared to state-of-the-art methods.