Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation

TL;DR

This paper introduces Causal-SAM-LLM, a novel framework that enhances medical image segmentation robustness by leveraging causal reasoning with large language models, reducing spurious correlations and enabling real-time, natural language-based corrections.

Contribution

It proposes a new causal reasoning framework for medical segmentation using LLMs, incorporating linguistic adversarial disentanglement and test-time causal intervention for improved out-of-distribution generalization.

Findings

State-of-the-art OOD robustness with up to 6.2 points Dice score improvement

Reduces Hausdorff Distance by 15.8 mm compared to baselines

Achieves this with less than 9% of trainable parameters

Abstract

The clinical utility of deep learning models for medical image segmentation is severely constrained by their inability to generalize to unseen domains. This failure is often rooted in the models learning spurious correlations between anatomical content and domain-specific imaging styles. To overcome this fundamental challenge, we introduce Causal-SAM-LLM, a novel framework that elevates Large Language Models (LLMs) to the role of causal reasoners. Our framework, built upon a frozen Segment Anything Model (SAM) encoder, incorporates two synergistic innovations. First, Linguistic Adversarial Disentanglement (LAD) employs a Vision-Language Model to generate rich, textual descriptions of confounding image styles. By training the segmentation model's features to be contrastively dissimilar to these style descriptions, it learns a representation robustly purged of non-causal information. Second, Test-Time Causal Intervention (TCI) provides an interactive mechanism where an LLM interprets a clinician's natural language command to modulate the segmentation decoder's features in real-time, enabling targeted error correction. We conduct an extensive empirical evaluation on a composite benchmark from four public datasets (BTCV, CHAOS, AMOS, BraTS), assessing generalization under cross-scanner, cross-modality, and cross-anatomy settings. Causal-SAM-LLM establishes a new state of the art in out-of-distribution (OOD) robustness, improving the average Dice score by up to 6.2 points and reducing the Hausdorff Distance by 15.8 mm over the strongest baseline, all while using less than 9% of the full model's trainable parameters. Our work charts a new course for building robust, efficient, and interactively controllable medical AI systems.