SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality

TL;DR

SimCMF introduces a straightforward cross-modal fine-tuning framework that adapts vision foundation models trained on RGB images to various other imaging modalities, significantly enhancing segmentation performance across different sensors.

Contribution

The paper proposes a novel cross-modal alignment module and constructs a new benchmark for evaluating performance transfer from RGB to other imaging modalities.

Findings

Improves segmentation mIoU from 22.15% to 53.88% on average across modalities.

Outperforms existing baseline methods in cross-modal transfer tasks.

Demonstrates the potential of foundation models in sensor data enhancement.

Abstract

Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework, SimCMF, to study an important problem: cross-modal fine-tuning from vision foundation models trained on natural RGB images to other imaging modalities of different physical properties (e.g., polarization). In SimCMF, we conduct a thorough analysis of different basic components from the most naive design and ultimately propose a novel cross-modal alignment module to address the modality misalignment problem. We apply SimCMF to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new imaging modality. Given the absence of relevant benchmarks, we construct a benchmark for performance evaluation. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. SimCMF can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. The code is available at https://github.com/mt-cly/SimCMF