AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder

TL;DR

AutoSAM adapts the Segment Anything Model for medical images by replacing its conditioning mechanism with an image encoder, achieving state-of-the-art results without additional fine-tuning.

Contribution

The paper introduces a new encoder that replaces SAM's conditioning, enabling effective medical image segmentation without fine-tuning SAM.

Findings

Achieves state-of-the-art results on medical image benchmarks.

Enables fully automatic segmentation without manual prompts.

Operates effectively without fine-tuning SAM.

Abstract

The recently introduced Segment Anything Model (SAM) combines a clever architecture and large quantities of training data to obtain remarkable image segmentation capabilities. However, it fails to reproduce such results for Out-Of-Distribution (OOD) domains such as medical images. Moreover, while SAM is conditioned on either a mask or a set of points, it may be desirable to have a fully automatic solution. In this work, we replace SAM's conditioning with an encoder that operates on the same input image. By adding this encoder and without further fine-tuning SAM, we obtain state-of-the-art results on multiple medical images and video benchmarks. This new encoder is trained via gradients provided by a frozen SAM. For inspecting the knowledge within it, and providing a lightweight segmentation solution, we also learn to decode it into a mask by a shallow deconvolution network.