IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark

TL;DR

IRGPT is a pioneering multi-modal large language model designed for real-world infrared images, utilizing a large-scale authentic dataset and a curriculum transfer learning strategy to outperform existing methods across multiple tasks.

Contribution

The paper introduces IRGPT, the first multi-modal large language model for infrared images, built on a large-scale authentic dataset and a novel curriculum transfer learning approach.

Findings

Achieves state-of-the-art results on 9 infrared vision tasks.

Utilizes a large-scale dataset with over 260K image-text pairs.

Demonstrates effective knowledge transfer from visible to infrared domains.

Abstract

Real-world infrared imagery presents unique challenges for vision-language models due to the scarcity of aligned text data and domain-specific characteristics. Although existing methods have advanced the field, their reliance on synthetic infrared images generated through style transfer from visible images, which limits their ability to capture the unique characteristics of the infrared modality. To address this, we propose IRGPT, the first multi-modal large language model for real-world infrared images, built upon a large-scale InfraRed-Text Dataset (IR-TD) comprising over 260K authentic image-text pairs. The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts, where the initial drafts originated from two complementary processes: (1) LLM-generated descriptions of visible images, and (2) rule-based descriptions of annotations. Furthermore, we introduce a bi-cross-modal curriculum transfer learning strategy that systematically transfers knowledge from visible to infrared domains by considering the difficulty scores of both infrared-visible and infrared-text. Evaluated on a benchmark of 9 tasks (e.g., recognition, grounding), IRGPT achieves state-of-the-art performance even compared with larger-scale models.