MTSIC: Multi-stage Transformer-based GAN for Spectral Infrared Image Colorization

TL;DR

This paper introduces MTSIC, a multi-stage Transformer-based GAN that leverages spectral information and self-attention mechanisms to improve the colorization of thermal infrared images, enhancing their visual quality and semantic accuracy.

Contribution

The paper presents a novel multi-stage Transformer GAN framework that integrates spectral self-attention and multi-scale wavelet blocks for superior infrared image colorization.

Findings

Outperforms traditional colorization methods in visual quality.

Effectively preserves spectral details and semantic information.

Reduces semantic ambiguity in infrared image colorization.

Abstract

Thermal infrared (TIR) images, acquired through thermal radiation imaging, are unaffected by variations in lighting conditions and atmospheric haze. However, TIR images inherently lack color and texture information, limiting downstream tasks and potentially causing visual fatigue. Existing colorization methods primarily rely on single-band images with limited spectral information and insufficient feature extraction capabilities, which often result in image distortion and semantic ambiguity. In contrast, multiband infrared imagery provides richer spectral data, facilitating the preservation of finer details and enhancing semantic accuracy. In this paper, we propose a generative adversarial network (GAN)-based framework designed to integrate spectral information to enhance the colorization of infrared images. The framework employs a multi-stage spectral self-attention Transformer network (MTSIC) as the generator. Each spectral feature is treated as a token for self-attention computation, and a multi-head self-attention mechanism forms a spatial-spectral attention residual block (SARB), achieving multi-band feature mapping and reducing semantic confusion. Multiple SARB units are integrated into a Transformer-based single-stage network (STformer), which uses a U-shaped architecture to extract contextual information, combined with multi-scale wavelet blocks (MSWB) to align semantic information in the spatial-frequency dual domain. Multiple STformer modules are cascaded to form MTSIC, progressively optimizing the reconstruction quality. Experimental results demonstrate that the proposed method significantly outperforms traditional techniques and effectively enhances the visual quality of infrared images.