CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding

TL;DR

CurConMix+ is a comprehensive spatio-temporal framework that enhances surgical workflow understanding by addressing class imbalance and semantic interdependence through contrastive learning and multi-scale temporal fusion, supported by a new hierarchical benchmark.

Contribution

It introduces CurConMix+, a novel hierarchical framework combining contrastive learning and temporal transformers, and LLS48, a new benchmark for complex laparoscopic procedures.

Findings

Outperforms state-of-the-art in triplet recognition

Demonstrates strong cross-level generalization

Provides a new hierarchical surgical dataset

Abstract

Surgical action triplet recognition aims to understand fine-grained surgical behaviors by modeling the interactions among instruments, actions, and anatomical targets. Despite its clinical importance for workflow analysis and skill assessment, progress has been hindered by severe class imbalance, subtle visual variations, and the semantic interdependence among triplet components. Existing approaches often address only a subset of these challenges rather than tackling them jointly, which limits their ability to form a holistic understanding. This study builds upon CurConMix, a spatial representation framework. At its core, a curriculum-guided contrastive learning strategy learns discriminative and progressively correlated features, further enhanced by structured hard-pair sampling and feature-level mixup. Its temporal extension, CurConMix+, integrates a Multi-Resolution Temporal Transformer (MRTT) that achieves robust, context-aware understanding by adaptively fusing multi-scale temporal features and dynamically balancing spatio-temporal cues. Furthermore, we introduce LLS48, a new, hierarchically annotated benchmark for complex laparoscopic left lateral sectionectomy, providing step-, task-, and action-level annotations. Extensive experiments on CholecT45 and LLS48 demonstrate that CurConMix+ not only outperforms state-of-the-art approaches in triplet recognition, but also exhibits strong cross-level generalization, as its fine-grained features effectively transfer to higher-level phase and step recognition tasks. Together, the framework and dataset provide a unified foundation for hierarchy-aware, reproducible, and interpretable surgical workflow understanding. The code and dataset will be publicly released on GitHub to facilitate reproducibility and further research.