From Phase Grounding to Intelligent Surgical Narratives

TL;DR

This paper introduces a CLIP-based multi-modal framework that automatically generates structured surgical timelines and narratives from videos, reducing manual annotation effort and improving surgical documentation.

Contribution

It presents a novel method that aligns surgical video frames with textual gesture descriptions using pretrained multi-modal models, enabling automatic timeline creation.

Findings

Effective alignment of video frames with gesture descriptions

Automatic generation of surgical timelines and narratives

Reduces manual annotation time for surgical videos

Abstract

Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.