FlowC2S: Flowing from Current to Succeeding Frames for Fast and Memory-Efficient Video Continuation

TL;DR

FlowC2S is a novel, efficient method for video continuation that directly flows from current to succeeding frames, improving speed and memory use while surpassing state-of-the-art results.

Contribution

The paper introduces FlowC2S, a new approach that fine-tunes a pre-trained flow model with optimal couplings and target inversion for superior video continuation.

Findings

Outperforms state-of-the-art scores in FID and FVD metrics.

Reduces model input dimensionality by flowing directly between frames.

Achieves high-quality video continuation with as few as five neural function evaluations.

Abstract

This paper introduces a novel methodology for generating fast and memory-efficient video continuations. Our method, dubbed FlowC2S, fine-tunes a pre-trained text-to-video flow model to learn a vector field between the current and succeeding video chunks. Two design choices are key. First, we introduce inherent optimal couplings, utilizing temporally adjacent video chunks during training as a practical proxy for true optimal couplings, resulting in straighter flows. Second, we incorporate target inversion, injecting the inverted latent of the target chunk into the input representation to strengthen correspondences and improve visual fidelity. By flowing directly from current to succeeding frames, instead of the common combination of current frames with noise to generate a video continuation, we reduce the dimensionality of the model input by a factor of two. The proposed method, fine-tuned from LTXV and Wan, surpasses the state-of-the-art scores across quantitative evaluations with FID and FVD, with as few as five neural function evaluations.