Scalpel-SAM: A Semi-Supervised Paradigm for Adapting SAM to Infrared Small Object Detection

TL;DR

This paper introduces Scalpel-SAM, a semi-supervised approach that adapts the Segment Anything Model (SAM) for infrared small object detection, effectively reducing annotation costs and improving detection performance.

Contribution

It presents a novel hierarchical MoE adapter and a two-stage knowledge transfer paradigm for semi-supervised IR small object detection using SAM.

Findings

Achieves comparable or superior performance with minimal annotations.

First semi-supervised paradigm addressing IR-SOT data scarcity with SAM.

Effective knowledge distillation and transfer for lightweight models.

Abstract

Infrared small object detection urgently requires semi-supervised paradigms due to the high cost of annotation. However, existing methods like SAM face significant challenges of domain gaps, inability of encoding physical priors, and inherent architectural complexity. To address this, we designed a Hierarchical MoE Adapter consisting of four white-box neural operators. Building upon this core component, we propose a two-stage paradigm for knowledge distillation and transfer: (1) Prior-Guided Knowledge Distillation, where we use our MoE adapter and 10% of available fully supervised data to distill SAM into an expert teacher (Scalpel-SAM); and (2) Deployment-Oriented Knowledge Transfer, where we use Scalpel-SAM to generate pseudo labels for training lightweight and efficient downstream models. Experiments demonstrate that with minimal annotations, our paradigm enables downstream models to achieve performance comparable to, or even surpassing, their fully supervised counterparts. To our knowledge, this is the first semi-supervised paradigm that systematically addresses the data scarcity issue in IR-SOT using SAM as the teacher model.