Learning to Color from Language

TL;DR

This paper introduces a language-conditioned image colorization method that allows users to control and manipulate the coloring of greyscale images through descriptive captions, improving accuracy and plausibility.

Contribution

It proposes two novel architectures for language-conditioned colorization, enabling more precise and user-controllable colorization of images based on natural language input.

Findings

Language-conditioned models outperform language-agnostic ones in colorization accuracy.

Manipulating captions effectively changes the colorization results.

The approach allows intuitive user control over image coloring.

Abstract

Automatic colorization is the process of adding color to greyscale images. We condition this process on language, allowing end users to manipulate a colorized image by feeding in different captions. We present two different architectures for language-conditioned colorization, both of which produce more accurate and plausible colorizations than a language-agnostic version. Through this language-based framework, we can dramatically alter colorizations by manipulating descriptive color words in captions.