Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

TL;DR

This paper introduces a Local Optimization method with Representation Continuity to accelerate autoregressive video generation training, reducing costs by half while maintaining quality.

Contribution

It proposes novel techniques—Local Optimization and Representation Continuity—to improve training efficiency and consistency in autoregressive video models.

Findings

Halves training cost without quality loss.

Local Optimization reduces error propagation.

Representation Continuity enhances video consistency.

Abstract

Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class- and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.