Prompt Tuning for Parameter-efficient Medical Image Segmentation

TL;DR

This paper introduces a prompt tuning approach for medical image segmentation that achieves near full fine-tuning performance with significantly fewer trainable parameters, using a frozen pre-trained UNet with class-dependent prompt tokens.

Contribution

It proposes a prompt-able UNet architecture with a novel self-supervision scheme, enabling effective parameter-efficient adaptation for medical image segmentation tasks.

Findings

Prompt tuning achieves performance close to full fine-tuning.

Only 0.85% of parameters are trainable, reducing computational cost.

Results show minimal performance gap compared to fully fine-tuned models.

Abstract

Neural networks pre-trained on a self-supervision scheme have become the standard when operating in data rich environments with scarce annotations. As such, fine-tuning a model to a downstream task in a parameter-efficient but effective way, e.g. for a new set of classes in the case of semantic segmentation, is of increasing importance. In this work, we propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets. Relying on the recently popularized prompt tuning approach, we provide a prompt-able UNet (PUNet) architecture, that is frozen after pre-training, but adaptable throughout the network by class-dependent learnable prompt tokens. We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes (contrastive prototype assignment, CPA) of a student teacher combination alongside a concurrent segmentation loss on a subset of classes. We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models on CT imaging datasets. As such, the difference between fully fine-tuned and prompt-tuned variants amounts to only 3.83 pp for the TCIA/BTCV dataset and 2.67 pp for the CT-ORG dataset in the mean Dice Similarity Coefficient (DSC, in %) while only prompt tokens, corresponding to 0.85% of the pre-trained backbone model with 6.8M frozen parameters, are adjusted. The code for this work is available on https://github.com/marcdcfischer/PUNet .