Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment

TL;DR

This paper presents a multi-modal self-supervised learning approach combining verbal feedback transcripts and surgical videos to automatically assess the effectiveness of surgical training feedback, improving prediction accuracy.

Contribution

It introduces a scalable, automated method integrating verbal and visual data for feedback effectiveness prediction, with self-supervised fine-tuning enhancing surgical video representations.

Findings

Combined modalities achieve AUROC of 0.70+/-0.02

Individual modalities are predictive of trainee behavior change

Self-supervised fine-tuning improves video representation and prediction performance

Abstract

During surgical training, real-time feedback from trainers to trainees is important for preventing errors and enhancing long-term skill acquisition. Accurately predicting the effectiveness of this feedback, specifically whether it leads to a change in trainee behavior, is crucial for developing methods for improving surgical training and education. However, relying on human annotations to assess feedback effectiveness is laborious and prone to biases, underscoring the need for an automated, scalable, and objective method. Creating such an automated system poses challenges, as it requires an understanding of both the verbal feedback delivered by the trainer and the visual context of the real-time surgical scene. To address this, we propose a method that integrates information from transcribed verbal feedback and corresponding surgical video to predict feedback effectiveness. Our findings show that both transcribed feedback and surgical video are individually predictive of trainee behavior changes, and their combination achieves an AUROC of 0.70+/-0.02, improving prediction accuracy by up to 6.6%. Additionally, we introduce self-supervised fine-tuning as a strategy for enhancing surgical video representation learning, which is scalable and further enhances prediction performance. Our results demonstrate the potential of multi-modal learning to advance the automated assessment of surgical feedback.