Toward Clinically Assisted Colorectal Polyp Recognition via Structured Cross-modal Representation Consistency

TL;DR

This paper introduces a novel Transformer-based method that uses structured cross-modal representation consistency to improve colorectal polyp classification accuracy in white-light images, eliminating the need for NBI images in clinical settings.

Contribution

The proposed approach aligns multi-modal features via a Spatial Attention Module, enabling accurate classification using only white-light images without relying on NBI images.

Findings

Outperforms recent methods in classification accuracy.

Achieves multi-modal prediction with a single Transformer.

Significantly improves WL image classification in clinical scenarios.

Abstract

The colorectal polyps classification is a critical clinical examination. To improve the classification accuracy, most computer-aided diagnosis algorithms recognize colorectal polyps by adopting Narrow-Band Imaging (NBI). However, the NBI usually suffers from missing utilization in real clinic scenarios since the acquisition of this specific image requires manual switching of the light mode when polyps have been detected by using White-Light (WL) images. To avoid the above situation, we propose a novel method to directly achieve accurate white-light colonoscopy image classification by conducting structured cross-modal representation consistency. In practice, a pair of multi-modal images, i.e. NBI and WL, are fed into a shared Transformer to extract hierarchical feature representations. Then a novel designed Spatial Attention Module (SAM) is adopted to calculate the similarities between the class token and patch tokens %from multi-levels for a specific modality image. By aligning the class tokens and spatial attention maps of paired NBI and WL images at different levels, the Transformer achieves the ability to keep both global and local representation consistency for the above two modalities. Extensive experimental results illustrate the proposed method outperforms the recent studies with a margin, realizing multi-modal prediction with a single Transformer while greatly improving the classification accuracy when only with WL images.