Channel-Aware Probing for Multi-Channel Imaging

TL;DR

This paper introduces Channel-Aware Probing (CAP), a novel method that leverages inter-channel diversity in multi-channel imaging data to improve the effectiveness of probing pre-trained vision encoders without full fine-tuning.

Contribution

The paper proposes CAP, which uses independent feature encoding and decoupled pooling to enhance probing of fixed encoders on MCI data, bridging the gap to full fine-tuning.

Findings

CAP improves probing performance across benchmarks.

CAP matches the results of training from scratch.

CAP reduces the gap to full fine-tuning.

Abstract

Training and evaluating vision encoders on Multi-Channel Imaging (MCI) data remains challenging as channel configurations vary across datasets, preventing fixed-channel training and limiting reuse of pre-trained encoders on new channel settings. Prior work trains MCI encoders but typically evaluates them via full fine-tuning, leaving probing with frozen pre-trained encoders comparatively underexplored. Existing studies that perform probing largely focus on improving representations, rather than how to best leverage fixed representations for downstream tasks. Although the latter problem has been studied in other domains, directly transferring those strategies to MCI yields weak results, even worse than training from scratch. We therefore propose Channel-Aware Probing (CAP), which exploits the intrinsic inter-channel diversity in MCI datasets by controlling feature flow at both the encoder and probe levels. CAP uses Independent Feature Encoding (IFE) to encode each channel separately, and Decoupled Pooling (DCP) to pool within channels before aggregating across channels. Across three MCI benchmarks, CAP consistently improves probing performance over the default probing protocol, matches fine-tuning from scratch, and largely reduces the gap to full fine-tuning from the same MCI pre-trained checkpoints. Code can be found in https://github.com/umarikkar/CAP.