SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

TL;DR

SurgMotion introduces a novel video-native foundation model for surgical videos that emphasizes semantic motion understanding over pixel-level details, achieving state-of-the-art results across multiple benchmarks.

Contribution

The paper presents SurgMotion, a new model with three innovations and a large-scale surgical video dataset, significantly improving surgical video analysis performance.

Findings

Outperforms state-of-the-art on surgical workflow recognition with up to 14.6% F1 score improvement.

Achieves 39.54% mAP-IVT on action triplet recognition.

Demonstrates effectiveness on skill assessment, polyp segmentation, and depth estimation.

Abstract

While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details, such as smoke, specular reflections, and fluid motion, rather than semantic structures essential for surgical understanding. We present SurgMotion, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), SurgMotion introduces three key technical innovations tailored to surgical videos: (1) motion-guided latent masked prediction to prioritize semantically meaningful regions, (2) spatiotemporal affinity self-distillation to enforce relational consistency, and (3) spatiotemporal feature diversity regularization (SFDR) to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate SurgMotion-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that SurgMotion significantly outperforms state-of-the-art methods on surgical workflow recognition, achieving 14.6 percent improvement in F1 score on EgoSurgery and 10.3 percent on PitVis; on action triplet recognition with 39.54 percent mAP-IVT on CholecT50; as well as on skill assessment, polyp segmentation, and depth estimation. These results establish SurgMotion as a new standard for universal, motion-oriented surgical video understanding.