ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room

TL;DR

This paper introduces a novel, sample-efficient object-based method for surgical activity recognition in operating rooms, emphasizing geometric interactions over image data, and demonstrates superior performance in low-data scenarios.

Contribution

The work presents a new object-centric approach that leverages geometric arrangements for activity recognition, reducing data requirements compared to traditional image-based methods.

Findings

Outperforms existing object-centric methods on clip-level classification

Effective in low-data regimes for long video activity recognition

Shows promise for real-time surgical activity monitoring

Abstract

Surgical robotics holds much promise for improving patient safety and clinician experience in the Operating Room (OR). However, it also comes with new challenges, requiring strong team coordination and effective OR management. Automatic detection of surgical activities is a key requirement for developing AI-based intelligent tools to tackle these challenges. The current state-of-the-art surgical activity recognition methods however operate on image-based representations and depend on large-scale labeled datasets whose collection is time-consuming and resource-expensive. This work proposes a new sample-efficient and object-based approach for surgical activity recognition in the OR. Our method focuses on the geometric arrangements between clinicians and surgical devices, thus utilizing the significant object interaction dynamics in the OR. We conduct experiments in a low-data regime study for long video activity recognition. We also benchmark our method againstother object-centric approaches on clip-level action classification and show superior performance.