Causality in Video Diffusers is Separable from Denoising

TL;DR

This paper demonstrates that causal reasoning in video diffusion models can be separated from the denoising process, leading to more efficient architectures without sacrificing quality.

Contribution

The paper introduces Separable Causal Diffusion (SCD), a novel architecture that decouples temporal reasoning from denoising, improving efficiency in video generation models.

Findings

SCD improves throughput and latency in video diffusion tasks.

Early layers in models show redundant features across denoising steps.

Deeper layers focus on intra-frame rendering with sparse cross-frame attention.

Abstract

Causality -- referring to temporal, uni-directional cause-effect relationships between components -- underlies many complex generative processes, including videos, language, and robot trajectories. Current causal diffusion models entangle temporal reasoning with iterative denoising, applying causal attention across all layers, at every denoising step, and over the entire context. In this paper, we show that the causal reasoning in these models is separable from the multi-step denoising process. Through systematic probing of autoregressive video diffusers, we uncover two key regularities: (1) early layers produce highly similar features across denoising steps, indicating redundant computation along the diffusion trajectory; and (2) deeper layers exhibit sparse cross-frame attention and primarily perform intra-frame rendering. Motivated by these findings, we introduce Separable Causal Diffusion (SCD), a new architecture that explicitly decouples once-per-frame temporal reasoning, via a causal transformer encoder, from multi-step frame-wise rendering, via a lightweight diffusion decoder. Extensive experiments on both pretraining and post-training tasks across synthetic and real benchmarks show that SCD significantly improves throughput and per-frame latency while matching or surpassing the generation quality of strong causal diffusion baselines.