M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision

TL;DR

M-SpecGene is a pioneering generalized foundation model for RGBT multispectral vision that learns modality-invariant representations through self-supervised pre-training, improving robustness and generalizability across diverse tasks and datasets.

Contribution

It introduces a unified, self-supervised framework for RGBT multispectral perception, addressing data and modality biases with novel metrics and training strategies.

Findings

Validated across eleven datasets and four tasks.

Achieved superior generalization compared to task-specific models.

Introduced the GMM-CMSS masking strategy for effective pre-training.

Abstract

RGB-Thermal (RGBT) multispectral vision is essential for robust perception in complex environments. Most RGBT tasks follow a case-by-case research paradigm, relying on manually customized models to learn task-oriented representations. Nevertheless, this paradigm is inherently constrained by artificial inductive bias, modality bias, and data bottleneck. To address these limitations, we make the initial attempt to build a Generalized RGBT MultiSpectral foundation model (M-SpecGene), which aims to learn modality-invariant representations from large-scale broad data in a self-supervised manner. M-SpecGene provides new insights into multispectral fusion and integrates prior case-by-case studies into a unified paradigm. Considering the unique characteristic of information imbalance in RGBT data, we introduce the Cross-Modality Structural Sparsity (CMSS) metric to quantify the information density across two modalities. Then we develop the GMM-CMSS progressive masking strategy to facilitate a flexible, easy-to-hard, and object-centric pre-training process. Comprehensive experiments validate M-SpecGene's generalizability across eleven datasets for four RGBT downstream tasks. The code will be available at https://github.com/CalayZhou/M-SpecGene.