OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

TL;DR

OmniForcing introduces a novel framework that enables real-time, high-quality joint audio-visual generation by distilling bidirectional diffusion models into an efficient autoregressive generator, overcoming latency and training stability issues.

Contribution

The paper presents the first streaming autoregressive framework for joint audio-visual generation, with innovative techniques to address training instability and synchronization challenges.

Findings

Achieves state-of-the-art streaming generation at ~25 FPS on a single GPU.

Maintains multi-modal synchronization and visual quality comparable to bidirectional models.

Introduces novel alignment and distillation techniques to stabilize training and improve performance.

Abstract

Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}