Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation

TL;DR

This paper introduces a novel affinity-guided convolutional network that effectively segments unpaired multimodal medical images by leveraging class-specific affinity matrices, enabling cross-modality learning without spatial alignment.

Contribution

It proposes a new affinity-guided network with class-specific affinity matrices for unpaired multimodal image segmentation, enhancing cross-modality generalization and hierarchical feature reasoning.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Effectively handles unpaired multimodal inputs without spatial alignment

Utilizes class-specific affinity matrices for hierarchical feature encoding

Abstract

Multi-modal medical image segmentation plays an essential role in clinical diagnosis. It remains challenging as the input modalities are often not well-aligned spatially. Existing learning-based methods mainly consider sharing trainable layers across modalities and minimizing visual feature discrepancies. While the problem is often formulated as joint supervised feature learning, multiple-scale features and class-specific representation have not yet been explored. In this paper, we propose an affinity-guided fully convolutional network for multimodal image segmentation. To learn effective representations, we design class-specific affinity matrices to encode the knowledge of hierarchical feature reasoning, together with the shared convolutional layers to ensure the cross-modality generalization. Our affinity matrix does not depend on spatial alignments of the visual features and thus allows us to train with unpaired, multimodal inputs. We extensively evaluated our method on two public multimodal benchmark datasets and outperform state-of-the-art methods.