Diffusing Colors: Image Colorization with Text Guided Diffusion

TL;DR

This paper introduces a novel image colorization method using diffusion models guided by text prompts, enhancing control, semantic accuracy, and visual quality in grayscale image colorization.

Contribution

The paper proposes a new diffusion-based colorization framework with text guidance and a CLIP-based ranking system, improving controllability and output quality over existing methods.

Findings

Outperforms existing colorization techniques in visual quality and semantic coherence.

Enables fine-tuned control over color vividness through text prompts.

Demonstrates effectiveness in color enhancement and historical image colorization.

Abstract

The colorization of grayscale images is a complex and subjective task with significant challenges. Despite recent progress in employing large-scale datasets with deep neural networks, difficulties with controllability and visual quality persist. To tackle these issues, we present a novel image colorization framework that utilizes image diffusion techniques with granular text prompts. This integration not only produces colorization outputs that are semantically appropriate but also greatly improves the level of control users have over the colorization process. Our method provides a balance between automation and control, outperforming existing techniques in terms of visual quality and semantic coherence. We leverage a pretrained generative Diffusion Model, and show that we can finetune it for the colorization task without losing its generative power or attention to text prompts. Moreover, we present a novel CLIP-based ranking model that evaluates color vividness, enabling automatic selection of the most suitable level of vividness based on the specific scene semantics. Our approach holds potential particularly for color enhancement and historical image colorization.