TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models

TL;DR

TalkingMachines is a real-time, audio-driven video synthesis framework that transforms pretrained models into natural conversational character animators, enabling seamless, high-quality video streaming driven by audio inputs.

Contribution

It adapts a pretrained image-to-video model into an audio-driven avatar generator and introduces efficient inference techniques for real-time performance.

Findings

Achieves real-time, high-quality audio-driven video synthesis

Enables infinite video streaming without error accumulation

Optimizes inference pipeline for low latency and high throughput

Abstract

In this paper, we present TalkingMachines -- an efficient framework that transforms pretrained video generation models into real-time, audio-driven character animators. TalkingMachines enables natural conversational experiences by integrating an audio large language model (LLM) with our video generation foundation model. Our primary contributions include: (1) We adapt a pretrained SOTA image-to-video DiT into an audio-driven avatar generation model of 18 billion parameters; (2) We enable infinite video streaming without error accumulation through asymmetric knowledge distillation from a bidirectional teacher model into a sparse causal, autoregressive student model; (3) We design a high-throughput, low-latency inference pipeline incorporating several key engineering optimizations such as: (a) disaggregation of the DiT and VAE decoder across separate devices, (b) efficient overlap of inter-device communication and computation using CUDA streams, (c) elimination of redundant recomputations to maximize frame-generation throughput. Please see demo videos here - https://aaxwaz.github.io/TalkingMachines/