Prompt-based Consistent Video Colorization

TL;DR

This paper introduces an automated, high-fidelity video colorization method that leverages language and segmentation guidance, optical flow for temporal consistency, and achieves state-of-the-art results without manual input.

Contribution

It presents a novel prompt-based approach combining language-conditioned diffusion models with automatic object masks and optical flow for consistent, high-quality video colorization.

Findings

Achieves state-of-the-art colorization accuracy (PSNR)

Demonstrates improved visual realism (Colorfulness, CDC)

Maintains temporal stability through flow-guided warping and correction

Abstract

Existing video colorization methods struggle with temporal flickering or demand extensive manual input. We propose a novel approach automating high-fidelity video colorization using rich semantic guidance derived from language and segmentation. We employ a language-conditioned diffusion model to colorize grayscale frames. Guidance is provided via automatically generated object masks and textual prompts; our primary automatic method uses a generic prompt, achieving state-of-the-art results without specific color input. Temporal stability is achieved by warping color information from previous frames using optical flow (RAFT); a correction step detects and fixes inconsistencies introduced by warping. Evaluations on standard benchmarks (DAVIS30, VIDEVO20) show our method achieves state-of-the-art performance in colorization accuracy (PSNR) and visual realism (Colorfulness, CDC), demonstrating the efficacy of automated prompt-based guidance for consistent video colorization.