SegPrompt: Using Segmentation Map as a Better Prompt to Finetune Deep Models for Kidney Stone Classification

TL;DR

SegPrompt leverages segmentation maps as prompts and auxiliary information to improve kidney stone classification with limited data, enhancing model performance and generalization.

Contribution

This paper introduces SegPrompt, a novel approach that uses segmentation maps both as additional features and prompts to fine-tune deep models efficiently.

Findings

SegPrompt outperforms baseline models in limited data scenarios.

SegPrompt reduces the number of trainable parameters compared to traditional fine-tuning.

SegPrompt achieves a better balance between fitting and generalization.

Abstract

Recently, deep learning has produced encouraging results for kidney stone classification using endoscope images. However, the shortage of annotated training data poses a severe problem in improving the performance and generalization ability of the trained model. It is thus crucial to fully exploit the limited data at hand. In this paper, we propose SegPrompt to alleviate the data shortage problems by exploiting segmentation maps from two aspects. First, SegPrompt integrates segmentation maps to facilitate classification training so that the classification model is aware of the regions of interest. The proposed method allows the image and segmentation tokens to interact with each other to fully utilize the segmentation map information. Second, we use the segmentation maps as prompts to tune the pretrained deep model, resulting in much fewer trainable parameters than vanilla finetuning. We perform extensive experiments on the collected kidney stone dataset. The results show that SegPrompt can achieve an advantageous balance between the model fitting ability and the generalization ability, eventually leading to an effective model with limited training data.