Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation

TL;DR

VL-DUN introduces a unified framework that jointly restores and segments medical images, leveraging the synergy between low-level signal recovery and high-level semantic understanding for improved accuracy and robustness.

Contribution

It formulates AiOMIRS as a joint optimization problem with an interpretable unfolding mechanism and a frequency-aware Mamba module, enabling efficient global context modeling and mutual task refinement.

Findings

Achieved state-of-the-art results on multi-modal benchmarks.

Improved PSNR by 0.92 dB and Dice coefficient by 9.76%.

Demonstrated robustness over isolated task processing.

Abstract

We propose VL-DUN, a principled framework for joint All-in-One Medical Image Restoration and Segmentation (AiOMIRS) that bridges the gap between low-level signal recovery and high-level semantic understanding. While standard pipelines treat these tasks in isolation, our core insight is that they are fundamentally synergistic: restoration provides clean anatomical structures to improve segmentation, while semantic priors regularize the restoration process. VL-DUN resolves the sub-optimality of sequential processing through two primary innovations. (1) We formulate AiOMIRS as a unified optimization problem, deriving an interpretable joint unfolding mechanism where restoration and segmentation are mathematically coupled for mutual refinement. (2) We introduce a frequency-aware Mamba mechanism to capture long-range dependencies for global segmentation while preserving the high-frequency textures necessary for restoration. This allows for efficient global context modeling with linear complexity, effectively mitigating the spectral bias of standard architectures. As a pioneering work in the AiOMIRS task, VL-DUN establishes a new state-of-the-art across multi-modal benchmarks, improving PSNR by 0.92 dB and the Dice coefficient by 9.76\%. Our results demonstrate that joint collaborative learning offers a superior, more robust solution for complex clinical workflows compared to isolated task processing. The codes are provided in https://github.com/cipi666/VLDUN.