Learning Collaborative Knowledge with Multimodal Representation for Polyp Re-Identification

TL;DR

This paper introduces a deep multimodal learning framework for colonoscopic polyp re-identification, improving retrieval accuracy by leveraging visual and text modalities and collaborative fusion strategies to address domain gaps.

Contribution

It proposes a novel DMCL framework with dynamic multimodal fusion, enhancing generalization and performance in medical polyp re-identification tasks.

Findings

Multimodal approach outperforms unimodal models on benchmarks.

Collaborative fusion improves retrieval accuracy.

End-to-end training effectively integrates visual and text features.

Abstract

Colonoscopic Polyp Re-Identification aims to match the same polyp from a large gallery with images from different views taken using different cameras, which plays an important role in the prevention and treatment of colorectal cancer in computer-aided diagnosis. However, traditional methods for object ReID directly adopting CNN models trained on the ImageNet dataset usually produce unsatisfactory retrieval performance on colonoscopic datasets due to the large domain gap. Worsely, these solutions typically learn unimodal modal representations on the basis of visual samples, which fails to explore complementary information from other different modalities. To address this challenge, we propose a novel Deep Multimodal Collaborative Learning framework named DMCL for polyp re-identification, which can effectively encourage multimodal knowledge collaboration and reinforce generalization capability in medical scenarios. On the basis of it, a dynamic multimodal feature fusion strategy is introduced to leverage the optimized visual-text representations for multimodal fusion via end-to-end training. Experiments on the standard benchmarks show the benefits of the multimodal setting over state-of-the-art unimodal ReID models, especially when combined with the collaborative multimodal fusion strategy. The code is publicly available at https://github.com/JeremyXSC/DMCL.