Hyper-Connected Transformer Network for Multi-Modality PET-CT Segmentation

TL;DR

This paper introduces a hyper-connected transformer network that effectively integrates multi-modality PET-CT images for improved tumor segmentation by capturing global dependencies and fusing complementary features.

Contribution

The study presents a novel hyper-connected transformer architecture with multiple branches and fusion mechanism for enhanced multi-modality image segmentation.

Findings

HCT outperforms existing methods in segmentation accuracy.

The hyper-connected fusion effectively combines complementary features.

Global dependencies are better captured with the transformer-based approach.

Abstract

[18F]-Fluorodeoxyglucose (FDG) positron emission tomography - computed tomography (PET-CT) has become the imaging modality of choice for diagnosing many cancers. Co-learning complementary PET-CT imaging features is a fundamental requirement for automatic tumor segmentation and for developing computer aided cancer diagnosis systems. In this study, we propose a hyper-connected transformer (HCT) network that integrates a transformer network (TN) with a hyper connected fusion for multi-modality PET-CT images. The TN was leveraged for its ability to provide global dependencies in image feature learning, which was achieved by using image patch embeddings with a self-attention mechanism to capture image-wide contextual information. We extended the single-modality definition of TN with multiple TN based branches to separately extract image features. We also introduced a hyper connected fusion to fuse the contextual and complementary image features across multiple transformers in an iterative manner. Our results with two clinical datasets show that HCT achieved better performance in segmentation accuracy when compared to the existing methods.