Taming Teacher Forcing for Masked Autoregressive Video Generation

TL;DR

MAGI is a novel hybrid video generation framework that uses Complete Teacher Forcing to improve autoregressive video synthesis, achieving state-of-the-art results and enabling long, coherent videos from limited data.

Contribution

The paper introduces Complete Teacher Forcing (CTF), a new training method that enhances autoregressive video generation by conditioning on complete frames, outperforming previous masked modeling approaches.

Findings

CTF improves FVD scores by 23% over MTF.

MAGI can generate videos longer than 100 frames.

Effective with as few as 16 training frames.

Abstract

We introduce MAGI, a hybrid video generation framework that combines masked modeling for intra-frame generation with causal modeling for next-frame generation. Our key innovation, Complete Teacher Forcing (CTF), conditions masked frames on complete observation frames rather than masked ones (namely Masked Teacher Forcing, MTF), enabling a smooth transition from token-level (patch-level) to frame-level autoregressive generation. CTF significantly outperforms MTF, achieving a +23% improvement in FVD scores on first-frame conditioned video prediction. To address issues like exposure bias, we employ targeted training strategies, setting a new benchmark in autoregressive video generation. Experiments show that MAGI can generate long, coherent video sequences exceeding 100 frames, even when trained on as few as 16 frames, highlighting its potential for scalable, high-quality video generation.