Context Forcing: Consistent Autoregressive Video Generation with Long Context

TL;DR

This paper introduces Context Forcing, a novel training framework for long video generation that uses a long-context teacher to improve global temporal consistency and extends effective context length well beyond current methods.

Contribution

It proposes a long-context training approach with a Slow-Fast Memory system to enable long-term video generation, addressing the student-teacher mismatch in existing methods.

Findings

Enables effective context lengths exceeding 20 seconds.

Outperforms state-of-the-art methods like LongLive and Infinite-RoPE.

Maintains superior long-term consistency in generated videos.

Abstract

Recent approaches to real-time long video generation typically employ streaming tuning strategies, attempting to train a long-context student using a short-context (memoryless) teacher. In these frameworks, the student performs long rollouts but receives supervision from a teacher limited to short 5-second windows. This structural discrepancy creates a critical \textbf{student-teacher mismatch}: the teacher's inability to access long-term history prevents it from guiding the student on global temporal dependencies, effectively capping the student's context length. To resolve this, we propose \textbf{Context Forcing}, a novel framework that trains a long-context student via a long-context teacher. By ensuring the teacher is aware of the full generation history, we eliminate the supervision mismatch, enabling the robust training of models capable of long-term consistency. To make this computationally feasible for extreme durations (e.g., 2 minutes), we introduce a context management system that transforms the linearly growing context into a \textbf{Slow-Fast Memory} architecture, significantly reducing visual redundancy. Extensive results demonstrate that our method enables effective context lengths exceeding 20 seconds -- 2 to 10 times longer than state-of-the-art methods like LongLive and Infinite-RoPE. By leveraging this extended context, Context Forcing preserves superior consistency across long durations, surpassing state-of-the-art baselines on various long video evaluation metrics.