TouchMap-OR: Multi-View 3D Mapping of Hand-Surface Contacts

TL;DR

TouchMap-OR is a multi-view RGB-D system that reconstructs detailed 3D hand-surface interactions in operating rooms, enabling precise contact history analysis to improve infection prevention.

Contribution

It introduces a novel multi-view 3D mapping system that models clinicians, hand geometry, and environment to accurately track and associate hand contacts with surfaces in medical settings.

Findings

Achieves 0.75 binary contact F1 score, outperforming baselines.

Maintains 0.96 identity attribution accuracy.

Reconstructs consistent multi-person 3D skeletons and hand meshes.

Abstract

Hand-surface interactions between clinicians, patients, and medical equipment play a central role in pathogen transmission during medical procedures. However, these interactions remain largely unobserved, as current infection-prevention practices rely on manual observation and cannot reconstruct detailed contact histories. In this work we formulate the problem of identity-resolved hand-surface interaction reconstruction in operating rooms and introduce TouchMap-OR, a multi-view RGB-D vision system that models clinicians, articulated hand geometry, and the semantic structure of the clinical environment to infer when and where contacts occur. The system reconstructs globally consistent multi-person 3D skeleton tracks across cameras while estimating articulated MANO hand meshes from RGB observations aligned to depth data. Multi-view hand reconstructions are fused and associated with tracked clinicians to obtain consistent left and right hand trajectories. A semantic 3D model of the operating room is built from multi-view segmentation and depth fusion, enabling reconstructed hand trajectories to be mapped to specific surfaces, including medical equipment, movable objects, and patient body sites. Temporal hand-surface proximity is used to infer contact episodes describing which clinician touched which surface and when. We evaluate TouchMap-OR on recordings from three real anesthesia inductions with manually annotated contact events. TouchMap-OR achieves 0.75 binary contact F1, outperforming tracking-based baselines while maintaining comparable multi-person tracking accuracy and achieving 0.96 identity attribution accuracy.