Semantic-driven Colorization

TL;DR

This paper introduces a semantic-driven colorization method that mimics human perception by first understanding the image's semantics before coloring, leading to more plausible and interpretable results.

Contribution

It proposes a novel approach that incorporates semantic understanding into the colorization process and redesigns the U-Net architecture with dual data streams for improved plausibility.

Findings

Provides plausible colors at a semantic level

Semantic information becomes understandable and interactive

Achieves competitive results for specific objects

Abstract

Recent colorization works implicitly predict the semantic information while learning to colorize black-and-white images. Consequently, the generated color is easier to be overflowed, and the semantic faults are invisible. As a human experience in colorization, our brains first detect and recognize the objects in the photo, then imagine their plausible colors based on many similar objects we have seen in real life, and finally colorize them, as described in the teaser. In this study, we simulate that human-like action to let our network first learn to understand the photo, then colorize it. Thus, our work can provide plausible colors at a semantic level. Plus, the semantic information of the learned model becomes understandable and able to interact. Additionally, we also prove that Instance Normalization is also a missing ingredient for colorization, then re-design the inference flow of U-Net to have two streams of data, providing an appropriate way of normalizing the feature maps from the black-and-white image and its semantic map. As a result, our network can provide plausible colors competitive to the typical colorization works for specific objects.