From Phase Grounding to Intelligent Surgical Narratives
Ethan Peterson, Huixin Zhan

TL;DR
This paper introduces a CLIP-based multi-modal framework that automatically generates structured surgical timelines and narratives from videos, reducing manual annotation effort and improving surgical documentation.
Contribution
It presents a novel method that aligns surgical video frames with textual gesture descriptions using pretrained multi-modal models, enabling automatic timeline creation.
Findings
Effective alignment of video frames with gesture descriptions
Automatic generation of surgical timelines and narratives
Reduces manual annotation time for surgical videos
Abstract
Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Human Pose and Action Recognition · Video Analysis and Summarization
