Articulated Clinician Detection Using 3D Pictorial Structures on RGB-D Data

TL;DR

This paper introduces a novel 3D pictorial structures approach utilizing RGB-D data for reliable clinician detection and pose estimation in operating rooms, overcoming challenges of lighting and occlusion.

Contribution

It extends the Pictorial Structures framework to 3D with new depth descriptors and inference methods, enabling effective clinician detection in complex surgical environments.

Findings

3D PS with RGB-D detectors outperforms 2D methods in challenging OR conditions

The histogram of depth differences (HDD) improves depth-based part detection

Exact inference in 3D PS enhances pose estimation accuracy

Abstract

Reliable human pose estimation (HPE) is essential to many clinical applications, such as surgical workflow analysis, radiation safety monitoring and human-robot cooperation. Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room. Instead, we propose a novel approach based on Pictorial Structures (PS) and on RGB-D data, which can be easily deployed in real ORs. We extend the PS framework in two ways. First, we build robust and discriminative part detectors using both color and depth images. We also present a novel descriptor for depth images, called histogram of depth differences (HDD). Second, we extend PS to 3D by proposing 3D pairwise constraints and a new method that makes exact inference tractable. Our approach is evaluated for pose estimation and clinician detection on a challenging RGB-D dataset recorded in a busy operating room during live surgeries. We conduct series of experiments to study the different part detectors in conjunction with the various 2D or 3D pairwise constraints. Our comparisons demonstrate that 3D PS with RGB-D part detectors significantly improves the results in a visually challenging operating environment.