Segment as You Wish -- Free-Form Language-Based Segmentation for Medical Images

TL;DR

This paper introduces FLanS, a novel medical image segmentation model that uses free-form natural language prompts and a symmetry-aware module to improve segmentation accuracy and robustness across diverse medical imaging scenarios.

Contribution

The paper presents FLanS, a new segmentation model capable of handling various free-form text prompts and ensuring consistent results across different scan orientations.

Findings

FLanS outperforms state-of-the-art baselines on multiple datasets.

The model demonstrates strong language understanding and segmentation accuracy.

Incorporates a symmetry-aware module for orientation-invariant segmentation.

Abstract

Medical imaging is crucial for diagnosing a patient's health condition, and accurate segmentation of these images is essential for isolating regions of interest to ensure precise diagnosis and treatment planning. Existing methods primarily rely on bounding boxes or point-based prompts, while few have explored text-related prompts, despite clinicians often describing their observations and instructions in natural language. To address this gap, we first propose a RAG-based free-form text prompt generator, that leverages the domain corpus to generate diverse and realistic descriptions. Then, we introduce FLanS, a novel medical image segmentation model that handles various free-form text prompts, including professional anatomy-informed queries, anatomy-agnostic position-driven queries, and anatomy-agnostic size-driven queries. Additionally, our model also incorporates a symmetry-aware canonicalization module to ensure consistent, accurate segmentations across varying scan orientations and reduce confusion between the anatomical position of an organ and its appearance in the scan. FLanS is trained on a large-scale dataset of over 100k medical images from 7 public datasets. Comprehensive experiments demonstrate the model's superior language understanding and segmentation precision, along with a deep comprehension of the relationship between them, outperforming SOTA baselines on both in-domain and out-of-domain datasets.