Towards Chunk-Wise Generation for Long Videos

TL;DR

This paper explores chunk-wise autoregressive methods for generating long videos, addressing memory constraints and inter-chunk consistency, and proposes a $k$-step search solution to improve long video synthesis.

Contribution

It provides a detailed survey of chunk-wise long video generation and introduces an efficient $k$-step search method to enhance inter-chunk coherence.

Findings

Chunk-wise autoregressive generation reduces memory load for long videos.

The $k$-step search improves consistency between video chunks.

Survey highlights challenges and solutions in long video synthesis.

Abstract

Generating long-duration videos has always been a significant challenge due to the inherent complexity of spatio-temporal domain and the substantial GPU memory demands required to calculate huge size tensors. While diffusion based generative models achieve state-of-the-art performance in video generation task, they are typically trained with predefined video resolutions and lengths. During inference, a noise tensor with specific resolution and length should be specified at first, and the model will perform denoising on the entire video tensor simultaneously, all the frames together. Such approach will easily raise an out-of-memory (OOM) problem when the specified resolution and/or length exceed a certain limit. One of the solutions to this problem is to generate many short video chunks autoregressively with strong inter-chunk spatio-temporal relation and then concatenate them together to form a long video. In this approach, a long video generation task is divided into multiple short video generation subtasks, and the cost of each subtask is reduced to a feasible level. In this paper, we conduct a detailed survey on long video generation with the autoregressive chunk-by-chunk strategy. We address common problems caused by applying short image-to-video models to long video tasks and design an efficient $k$-step search solution to mitigate these problems.