Not Quite Anything: Overcoming SAMs Limitations for 3D Medical Imaging

TL;DR

This paper introduces a compositional approach that enhances foundation models for 3D medical imaging by combining their outputs with MRI data, achieving high accuracy without retraining, and demonstrating its effectiveness in pediatric OCD studies.

Contribution

It proposes a novel method that treats foundation model outputs as additional input channels, improving segmentation accuracy in brain MRI without fine-tuning the models.

Findings

Achieves about 96% volume accuracy in basal ganglia segmentation.

Fast, label-efficient, and robust to out-of-distribution scans.

Effective in studying inflammation-related changes in pediatric OCD.

Abstract

Foundation segmentation models such as SAM and SAM-2 perform well on natural images but struggle with brain MRIs where structures like the caudate and thalamus lack sharp boundaries and have low contrast. Rather than fine tune these models (for example MedSAM), we propose a compositional alternative where the foundation model output is treated as an additional input channel and passed alongside the MRI to highlight regions of interest. We generate SAM-2 prompts by using a lightweight 3D U-Net that was previously trained on MRI segmentation. The U-Net may have been trained on a different dataset, so its guesses are often imprecise but usually in the correct region. The edges of the resulting foundation model guesses are smoothed to improve alignment with the MRI. We also test prompt free segmentation using DINO attention maps in the same framework. This has-a architecture avoids modifying foundation weights and adapts to domain shift without retraining the foundation model. It reaches about 96 percent volume accuracy on basal ganglia segmentation, which is sufficient for our study of longitudinal volume change. The approach is fast, label efficient, and robust to out of distribution scans. We apply it to study inflammation linked changes in sudden onset pediatric OCD.