Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior

TL;DR

This paper introduces a novel multimodal diffusion-based pipeline for automatic image colorization that enhances color saturation, semantic accuracy, and visual quality, outperforming previous methods in realism and user preference.

Contribution

It proposes a diffusion prior with luminance guidance and multimodal semantic priors to improve colorization quality and realism.

Findings

Outperforms previous methods in perceptual realism.

Generates saturated, semantically plausible colors.

Surpasses prior approaches in human preference studies.

Abstract

Colorizing grayscale images offers an engaging visual experience. Existing automatic colorization methods often fail to generate satisfactory results due to incorrect semantic colors and unsaturated colors. In this work, we propose an automatic colorization pipeline to overcome these challenges. We leverage the extraordinary generative ability of the diffusion prior to synthesize color with plausible semantics. To overcome the artifacts introduced by the diffusion prior, we apply the luminance conditional guidance. Moreover, we adopt multimodal high-level semantic priors to help the model understand the image content and deliver saturated colors. Besides, a luminance-aware decoder is designed to restore details and enhance overall visual quality. The proposed pipeline synthesizes saturated colors while maintaining plausible semantics. Experiments indicate that our proposed method considers both diversity and fidelity, surpassing previous methods in terms of perceptual realism and gain most human preference.