L-C4: Language-Based Video Colorization for Creative and Consistent Color

TL;DR

L-C4 is a novel language-guided video colorization method that enables creative, semantically accurate, and temporally consistent coloring by leveraging a pre-trained cross-modality generative model and innovative attention mechanisms.

Contribution

It introduces a cross-modality pre-fusion module, temporally deformable attention, and cross-clip fusion to improve creative control and temporal consistency in video colorization.

Findings

Outperforms existing methods in semantic accuracy and creativity.

Achieves stable, flicker-free colorization across video frames.

Demonstrates robustness in maintaining long-term color consistency.

Abstract

Automatic video colorization is inherently an ill-posed problem because each monochrome frame has multiple optional color candidates. Previous exemplar-based video colorization methods restrict the user's imagination due to the elaborate retrieval process. Alternatively, conditional image colorization methods combined with post-processing algorithms still struggle to maintain temporal consistency. To address these issues, we present Language-based video Colorization for Creative and Consistent Colors (L-C4) to guide the colorization process using user-provided language descriptions. Our model is built upon a pre-trained cross-modality generative model, leveraging its comprehensive language understanding and robust color representation abilities. We introduce the cross-modality pre-fusion module to generate instance-aware text embeddings, enabling the application of creative colors. Additionally, we propose temporally deformable attention to prevent flickering or color shifts, and cross-clip fusion to maintain long-term color consistency. Extensive experimental results demonstrate that L-C4 outperforms relevant methods, achieving semantically accurate colors, unrestricted creative correspondence, and temporally robust consistency.