A vision-language model and platform for temporally mapping surgery from video

TL;DR

This paper introduces Halsted, a comprehensive vision-language model trained on a large surgical video dataset, enabling automatic, accurate mapping of surgical procedures to support clinical use and autonomous robotic surgery.

Contribution

The paper presents Halsted, a novel, scalable vision-language model trained on the extensive Halsted Surgical Atlas, with a publicly released subset for benchmarking, advancing surgical AI capabilities.

Findings

Halsted outperforms previous models in surgical activity mapping.

The model offers greater comprehensiveness and efficiency.

The Halsted platform enables surgeons to map procedures within minutes.

Abstract

Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery. Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from videos, yet current models remain narrow in scope, capturing limited behavioural components within single procedures, and offer limited translational value, as they remain inaccessible to practising surgeons. Here we introduce Halsted, a vision-language model trained on the Halsted Surgical Atlas (HSA), one of the most comprehensive annotated video libraries grown through an iterative self-labelling framework and encompassing over 650,000 videos across eight surgical specialties. To facilitate benchmarking, we publicly release HSA-27k, a subset of the Halsted Surgical Atlas. Halsted surpasses previous state-of-the-art models in mapping surgical activity while offering greater comprehensiveness and computational efficiency. To bridge the longstanding translational gap of surgical AI, we develop the Halsted web platform (https://halstedhealth.ai/) to provide surgeons anywhere in the world with the previously-unavailable capability of automatically mapping their own procedures within minutes. By standardizing unstructured surgical video data and making these capabilities directly accessible to surgeons, our work brings surgical AI closer to clinical deployment and helps pave the way toward autonomous robotic surgery.