Efficient In-Context Medical Segmentation with Meta-driven Visual Prompt Selection

TL;DR

This paper introduces a meta-learning based visual prompt selection method for medical image segmentation, enhancing generalization and efficiency of large vision models without fine-tuning.

Contribution

It proposes a novel Meta-driven Visual Prompt Selection (MVPS) mechanism that actively chooses optimal prompts to improve medical segmentation performance.

Findings

Consistent performance improvements across 8 datasets and 4 tasks.

Enhanced generalization and efficiency in medical image segmentation.

Flexible, finetuning-free prompt selection module.

Abstract

In-context learning (ICL) with Large Vision Models (LVMs) presents a promising avenue in medical image segmentation by reducing the reliance on extensive labeling. However, the ICL performance of LVMs highly depends on the choices of visual prompts and suffers from domain shifts. While existing works leveraging LVMs for medical tasks have focused mainly on model-centric approaches like fine-tuning, we study an orthogonal data-centric perspective on how to select good visual prompts to facilitate generalization to medical domain. In this work, we propose a label-efficient in-context medical segmentation method by introducing a novel Meta-driven Visual Prompt Selection mechanism (MVPS), where a prompt retriever obtained from a meta-learning framework actively selects the optimal images as prompts to promote model performance and generalizability. Evaluated on 8 datasets and 4 tasks across 3 medical imaging modalities, our proposed approach demonstrates consistent gains over existing methods under different scenarios, improving both computational and label efficiency. Finally, we show that MVPS is a flexible, finetuning-free module that could be easily plugged into different backbones and combined with other model-centric approaches.