On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training

TL;DR

This study empirically demonstrates that incorporating depth information during pre-training significantly enhances surgical vision models' performance and data efficiency without altering inference architecture.

Contribution

It provides the first large-scale empirical comparison of RGB versus RGB-D pre-training for surgical vision foundation models, highlighting the benefits of geometric-aware pre-training.

Findings

Depth-aware models outperform RGB-only models across tasks.

Geometric pre-training improves data efficiency, surpassing full-data RGB models with only 25% data.

Depth is used only during pre-training, simplifying practical adoption.

Abstract

Vision foundation models (VFMs) have emerged as powerful tools for surgical scene understanding. However, current approaches predominantly rely on unimodal RGB pre-training, overlooking the complex 3D geometry inherent to surgical environments. Although several architectures support multimodal or geometry-aware inputs in general computer vision, the benefits of incorporating depth information in surgical settings remain underexplored. We conduct a large-scale empirical study comparing eight ViT-based VFMs that differ in pre-training domain, learning objective, and input modality (RGB vs. RGB-D). For pre-training, we use a curated dataset of 1.4 million robotic surgical images paired with depth maps generated from an off-the-shelf network. We evaluate these models under both frozen-backbone and end-to-end fine-tuning protocols across eight surgical datasets spanning object detection, segmentation, depth estimation, and pose estimation. Our experiments yield several consistent findings. Models incorporating explicit geometric tokenization, such as MultiMAE, substantially outperform unimodal baselines across all tasks. Notably, geometric-aware pre-training enables remarkable data efficiency: models fine-tuned on just 25% of labeled data consistently surpass RGB-only models trained on the full dataset. Importantly, these gains require no architectural or runtime changes at inference; depth is used only during pre-training, making adoption straightforward. These findings suggest that multimodal pre-training offers a viable path towards building more capable surgical vision systems.