Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning

TL;DR

This paper introduces Colon-X, a comprehensive multimodal dataset and models for colonoscopy, emphasizing the transition from understanding to clinical reasoning, and demonstrating improved reasoning accuracy in colonoscopy analysis.

Contribution

The paper presents ColonVQA, the largest multimodal colonoscopy dataset, and develops ColonR1, a reasoning-focused model that outperforms supervised methods under data scarcity.

Findings

Leading multimodal models lack robustness and trustworthiness.

ColonR1 achieves 56.61% accuracy, surpassing supervised fine-tuning by 25.22%.

Provides publicly available data and models for colonoscopy AI research.

Abstract

In this study, we present Colon-X, an open initiative aimed at advancing multimodal intelligence in colonoscopy. We begin by constructing ColonVQA, the most comprehensive multimodal dataset ever built for colonoscopy, featuring over 1.1M+ visual question answering entries across 76 clinical findings and 18 multimodal tasks. Beyond serving as a community-wide data foundation, we further investigate a critical yet underexplored transition in colonoscopy - evolving from multimodal understanding to clinical reasoning: (a) To capture the current landscape of multimodal understanding behaviors, we systematically assess the generalizability of 22 multimodal large language models and examine their reliability under human-induced perturbations. The results reveal that clinical outputs from leading MLLMs remain far from robust and trustworthy. (b) To narrow this gap, we further explore reasoning-centric intelligence tailored for colonoscopy. Specifically, we curate ColonReason, a clinically grounded reasoning dataset annotated through a multi-agent debating pipeline, and develop ColonR1, the first R1-styled model that mitigates reward information collapse through task-adaptive rewards and gradient-stable policy optimization. Under data-scarce conditions, our ColonR1 achieves 56.61% overall accuracy, outperforming supervised fine-tuning by 25.22%, and sets a new reasoning-enabled baseline for multimodal colonoscopy analysis. All data and model resources are publicly available at https://github.com/ai4colonoscopy/Colon-X.