EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopy

TL;DR

EndoVLA is a specialized vision-language-action model designed for autonomous endoscopic tracking, improving robustness and generalization in complex GI environments through dual-phase training.

Contribution

We introduce EndoVLA, a novel dual-phase VLA model tailored for robotic endoscopy, addressing data scarcity and domain shifts for improved autonomous tracking.

Findings

Enhanced tracking accuracy in endoscopy

Zero-shot generalization to diverse scenes

Effective handling of complex sequential tasks

Abstract

In endoscopic procedures, autonomous tracking of abnormal regions and following circumferential cutting markers can significantly reduce the cognitive burden on endoscopists. However, conventional model-based pipelines are fragile for each component (e.g., detection, motion planning) requires manual tuning and struggles to incorporate high-level endoscopic intent, leading to poor generalization across diverse scenes. Vision-Language-Action (VLA) models, which integrate visual perception, language grounding, and motion planning within an end-to-end framework, offer a promising alternative by semantically adapting to surgeon prompts without manual recalibration. Despite their potential, applying VLA models to robotic endoscopy presents unique challenges due to the complex and dynamic anatomical environments of the gastrointestinal (GI) tract. To address this, we introduce EndoVLA, designed specifically for continuum robots in GI interventions. Given endoscopic images and surgeon-issued tracking prompts, EndoVLA performs three core tasks: (1) polyp tracking, (2) delineation and following of abnormal mucosal regions, and (3) adherence to circular markers during circumferential cutting. To tackle data scarcity and domain shifts, we propose a dual-phase strategy comprising supervised fine-tuning on our EndoVLA-Motion dataset and reinforcement fine-tuning with task-aware rewards. Our approach significantly improves tracking performance in endoscopy and enables zero-shot generalization in diverse scenes and complex sequential tasks.