Efficient Continuous Video Flow Model for Video Prediction

TL;DR

This paper introduces an efficient continuous video flow model that significantly reduces latency and computational costs in multi-step video prediction, achieving state-of-the-art results on multiple datasets.

Contribution

The paper presents a novel modeling approach that decreases sample steps and model size for improved video prediction efficiency.

Findings

Reduces prediction latency by decreasing sample steps.

Minimizes model size to one-third of original.

Achieves state-of-the-art performance on multiple datasets.

Abstract

Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.