SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer

TL;DR

SeATrans leverages Transformer-based segmentation knowledge transfer to improve disease diagnosis accuracy across various medical imaging tasks.

Contribution

The paper introduces SeATrans, a novel Transformer-based model that dynamically integrates multi-scale segmentation features into diagnosis models, surpassing existing methods.

Findings

SeATrans outperforms state-of-the-art segmentation-assisted diagnosis methods.

The asymmetric multi-scale interaction enhances feature correlation.

SeA-block effectively vitalizes diagnosis features with segmentation information.

Abstract

Clinically, the accurate annotation of lesions/tissues can significantly facilitate the disease diagnosis. For example, the segmentation of optic disc/cup (OD/OC) on fundus image would facilitate the glaucoma diagnosis, the segmentation of skin lesions on dermoscopic images is helpful to the melanoma diagnosis, etc. With the advancement of deep learning techniques, a wide range of methods proved the lesions/tissues segmentation can also facilitate the automated disease diagnosis models. However, existing methods are limited in the sense that they can only capture static regional correlations in the images. Inspired by the global and dynamic nature of Vision Transformer, in this paper, we propose Segmentation-Assisted diagnosis Transformer (SeATrans) to transfer the segmentation knowledge to the disease diagnosis network. Specifically, we first propose an asymmetric multi-scale interaction strategy to correlate each single low-level diagnosis feature with multi-scale segmentation features. Then, an effective strategy called SeA-block is adopted to vitalize diagnosis feature via correlated segmentation features. To model the segmentation-diagnosis interaction, SeA-block first embeds the diagnosis feature based on the segmentation information via the encoder, and then transfers the embedding back to the diagnosis feature space by a decoder. Experimental results demonstrate that SeATrans surpasses a wide range of state-of-the-art (SOTA) segmentation-assisted diagnosis methods on several disease diagnosis tasks.