Harnessing Foundation Models for Robust and Generalizable 6-DOF Bronchoscopy Localization

TL;DR

This paper introduces PANSv2, a novel framework for 6-DOF bronchoscopy localization that combines multiple visual cues and foundation models to achieve high accuracy, robustness, and generalization across patient cases.

Contribution

PANSv2 integrates depth estimation, landmark detection, and centerline constraints with foundation models for improved bronchoscopy localization.

Findings

Achieves 18.1% improvement in SR-5 accuracy over existing methods.

Demonstrates high success rate across 10 patient cases.

Enhances robustness with automatic re-initialization during visual degradation.

Abstract

Vision-based 6-DOF bronchoscopy localization offers a promising solution for accurate and cost-effective interventional guidance. However, existing methods struggle with 1) limited generalization across patient cases due to scarce labeled data, and 2) poor robustness under visual degradation, as bronchoscopy procedures frequently involve artifacts such as occlusions and motion blur that impair visual information. To address these challenges, we propose PANSv2, a generalizable and robust bronchoscopy localization framework. Motivated by PANS that leverages multiple visual cues for pose likelihood measurement, PANSv2 integrates depth estimation, landmark detection, and centerline constraints into a unified pose optimization framework that evaluates pose probability and solves for the optimal bronchoscope pose. To further enhance generalization capabilities, we leverage the endoscopic foundation model EndoOmni for depth estimation and the video foundation model EndoMamba for landmark detection, incorporating both spatial and temporal analyses. Pretrained on diverse endoscopic datasets, these models provide stable and transferable visual representations, enabling reliable performance across varied bronchoscopy scenarios. Additionally, to improve robustness to visual degradation, we introduce an automatic re-initialization module that detects tracking failures and re-establishes pose using landmark detections once clear views are available. Experimental results on bronchoscopy dataset encompassing 10 patient cases show that PANSv2 achieves the highest tracking success rate, with an 18.1% improvement in SR-5 (percentage of absolute trajectory error under 5 mm) compared to existing methods, showing potential towards real clinical usage.