MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation

TL;DR

MirrorMe is a real-time, high-fidelity audio-driven half-body animation framework that leverages a diffusion transformer model with novel identity, audio, and training innovations for improved temporal coherence and gesture control.

Contribution

The paper introduces MirrorMe, a novel real-time framework using diffusion transformers with new mechanisms for identity preservation, audio synchronization, and multi-level training for high-quality animated videos.

Findings

State-of-the-art fidelity and lip-sync accuracy

Enhanced temporal stability in animations

Effective gesture control including hand poses

Abstract

Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.