CapsDT: Diffusion-Transformer for Capsule Robot Manipulation

TL;DR

CapsDT is a novel Diffusion Transformer model that integrates visual and textual data to improve capsule robot manipulation in endoscopy, demonstrating state-of-the-art performance in simulated and real-world tasks.

Contribution

This work introduces CapsDT, the first diffusion-transformer-based model for capsule robot control using vision and language inputs, enhancing endoscopy task performance.

Findings

Achieves state-of-the-art results in simulated endoscopy tasks.

Attains 26.25% success rate in real-world capsule manipulation.

Demonstrates robustness across various endoscopy scenarios.

Abstract

Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation.