HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition

TL;DR

HecVL introduces a hierarchical video-language pretraining method that leverages multi-level textual supervision to enable zero-shot and cross-procedure surgical phase recognition, enhancing model transferability and reducing annotation needs.

Contribution

The paper proposes a novel hierarchical pretraining framework with a multi-level dataset and contrastive learning, improving zero-shot surgical phase recognition and cross-dataset transferability.

Findings

Enables zero-shot surgical phase recognition without human annotations.

Achieves effective transfer across different surgical procedures and medical centers.

Disentangles hierarchical embeddings to encode both short-term and long-term surgical concepts.

Abstract

Natural language could play an important role in developing generalist surgical models by providing a broad source of supervision from raw texts. This flexible form of supervision can enable the model's transferability across datasets and tasks as natural language can be used to reference learned visual concepts or describe new ones. In this work, we present HecVL, a novel hierarchical video-language pretraining approach for building a generalist surgical model. Specifically, we construct a hierarchical video-text paired dataset by pairing the surgical lecture video with three hierarchical levels of texts: at clip-level, atomic actions using transcribed audio texts; at phase-level, conceptual text summaries; and at video-level, overall abstract text of the surgical procedure. Then, we propose a novel fine-to-coarse contrastive learning framework that learns separate embedding spaces for the three video-text hierarchies using a single model. By disentangling embedding spaces of different hierarchical levels, the learned multi-modal representations encode short-term and long-term surgical concepts in the same model. Thanks to the injected textual semantics, we demonstrate that the HecVL approach can enable zero-shot surgical phase recognition without any human annotation. Furthermore, we show that the same HecVL model for surgical phase recognition can be transferred across different surgical procedures and medical centers. The code is available at https://github.com/CAMMA-public/SurgVLP