TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot

TL;DR

This paper introduces TongueSAM, a universal tongue segmentation model based on SAM that leverages zero-shot learning to achieve high accuracy across diverse and challenging tongue images, marking the first application of large-scale pretrained models in this domain.

Contribution

The paper presents TongueSAM, a novel zero-shot tongue segmentation model based on SAM with an integrated Prompt Generator, demonstrating superior performance on various datasets.

Findings

Achieves 95.23% mIoU in zero-shot tongue segmentation

Outperforms existing methods on challenging background images

First application of large-scale pretrained model for tongue segmentation

Abstract

Tongue segmentation serves as the primary step in automated TCM tongue diagnosis, which plays a significant role in the diagnostic results. Currently, numerous deep learning based methods have achieved promising results. However, when confronted with tongue images that differ from the training set or possess challenging backgrounds, these methods demonstrate limited performance. To address this issue, this paper proposes a universal tongue segmentation model named TongueSAM based on SAM (Segment Anything Model). SAM is a large-scale pretrained interactive segmentation model known for its powerful zero-shot generalization capability. Applying SAM to tongue segmentation leverages its learned prior knowledge from natural images, enabling the achievement of zero-shot segmentation for various types of tongue images. In this study, a Prompt Generator based on object detection is integrated into SAM to enable an end-to-end automated tongue segmentation method. Experiments demonstrate that TongueSAM achieves exceptional performance across various of tongue segmentation datasets, particularly under zero-shot. Even when dealing with challenging background tongue images, TongueSAM achieves a mIoU of 95.23\% under zero-shot conditions, surpassing other segmentation methods. As far as we know, this is the first application of large-scale pretrained model for tongue segmentation. The project mentioned in this paper is currently publicly available.