Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuning

TL;DR

This paper introduces Video Consistency Distance (VCD), a novel frequency-domain metric for improving temporal consistency in image-to-video generation, and demonstrates its effectiveness through reward-based fine-tuning without sacrificing quality.

Contribution

The paper proposes VCD, a new frequency-space metric for temporal consistency, and applies it in reward-based fine-tuning to enhance video coherence in I2V tasks.

Findings

VCD significantly improves temporal consistency in generated videos.

Fine-tuning with VCD does not degrade overall video quality.

Experimental results validate VCD's effectiveness across multiple datasets.

Abstract

Reward-based fine-tuning of video diffusion models is an effective approach to improve the quality of generated videos, as it can fine-tune models without requiring real-world video datasets. However, it can sometimes be limited to specific performances because conventional reward functions are mainly aimed at enhancing the quality across the whole generated video sequence, such as aesthetic appeal and overall consistency. Notably, the temporal consistency of the generated video often suffers when applying previous approaches to image-to-video (I2V) generation tasks. To address this limitation, we propose Video Consistency Distance (VCD), a novel metric designed to enhance temporal consistency, and fine-tune a model with the reward-based fine-tuning framework. To achieve coherent temporal consistency relative to a conditioning image, VCD is defined in the frequency space of video frame features to capture frame information effectively through frequency-domain analysis. Experimental results across multiple I2V datasets demonstrate that fine-tuning a video generation model with VCD significantly enhances temporal consistency without degrading other performance compared to the previous method.