Cooperative Colorization: Exploring Latent Cross-Domain Priors for NIR Image Spectrum Translation

TL;DR

This paper introduces CoColor, a cooperative learning framework for NIR image spectrum translation that leverages latent cross-domain priors and bilateral domain translation to improve colorization quality and generalization.

Contribution

The paper proposes a novel cooperative learning paradigm with bilateral domain translation and latent priors, enhancing NIR-RGB spectrum translation performance.

Findings

Outperforms state-of-the-art methods by 3.95dB in PNSR for NIR colorization.

Achieves more diverse colors and richer textures in translated images.

Demonstrates effective domain knowledge exchange through intermittent transformations.

Abstract

Near-infrared (NIR) image spectrum translation is a challenging problem with many promising applications. Existing methods struggle with the mapping ambiguity between the NIR and the RGB domains, and generalize poorly due to the limitations of models' learning capabilities and the unavailability of sufficient NIR-RGB image pairs for training. To address these challenges, we propose a cooperative learning paradigm that colorizes NIR images in parallel with another proxy grayscale colorization task by exploring latent cross-domain priors (i.e., latent spectrum context priors and task domain priors), dubbed CoColor. The complementary statistical and semantic spectrum information from these two task domains -- in the forms of pre-trained colorization networks -- are brought in as task domain priors. A bilateral domain translation module is subsequently designed, in which intermittent NIR images are generated from grayscale and colorized in parallel with authentic NIR images; and vice versa for the grayscale images. These intermittent transformations act as latent spectrum context priors for efficient domain knowledge exchange. We progressively fine-tune and fuse these modules with a series of pixel-level and feature-level consistency constraints. Experiments show that our proposed cooperative learning framework produces satisfactory spectrum translation outputs with diverse colors and rich textures, and outperforms state-of-the-art counterparts by 3.95dB and 4.66dB in terms of PNSR for the NIR and grayscale colorization tasks, respectively.