VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks

TL;DR

VLSM-Adapter is a lightweight, transformer-based adapter that enables efficient fine-tuning of vision-language segmentation models, achieving high performance with significantly fewer trainable parameters, especially beneficial for medical imaging tasks.

Contribution

We propose a novel VLSM-Adapter that allows effective fine-tuning of pretrained models with minimal parameters, reducing computational costs while maintaining state-of-the-art performance.

Findings

Outperforms existing methods with only 3 million trainable parameters.

Achieves comparable results to full fine-tuning on CLIP-based segmentation models.

Reduces resource requirements for medical image segmentation tasks.

Abstract

Foundation Vision-Language Models (VLMs) trained using large-scale open-domain images and text pairs have recently been adapted to develop Vision-Language Segmentation Models (VLSMs) that allow providing text prompts during inference to guide image segmentation. If robust and powerful VLSMs can be built for medical images, it could aid medical professionals in many clinical tasks where they must spend substantial time delineating the target structure of interest. VLSMs for medical images resort to fine-tuning base VLM or VLSM pretrained on open-domain natural image datasets due to fewer annotated medical image datasets; this fine-tuning is resource-consuming and expensive as it usually requires updating all or a significant fraction of the pretrained parameters. Recently, lightweight blocks called adapters have been proposed in VLMs that keep the pretrained model frozen and only train adapters during fine-tuning, substantially reducing the computing resources required. We introduce a novel adapter, VLSM-Adapter, that can fine-tune pretrained vision-language segmentation models using transformer encoders. Our experiments in widely used CLIP-based segmentation models show that with only 3 million trainable parameters, the VLSM-Adapter outperforms state-of-the-art and is comparable to the upper bound end-to-end fine-tuning. The source code is available at: https://github.com/naamiinepal/vlsm-adapter.