Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video

TL;DR

Patient4D introduces a novel pipeline that leverages stationarity priors and geometric constraints to improve 3D patient body mesh recovery from monocular surgical videos, addressing occlusion and camera movement challenges.

Contribution

The paper presents Patient4D, a new method combining perception models with geometric mechanisms to enhance temporal consistency and robustness in monocular 3D mesh recovery during surgery.

Findings

Achieves 0.75 mean IoU under occlusion, significantly reducing failure frames.

Outperforms existing methods on synthetic and public benchmarks.

Demonstrates effectiveness of stationarity priors in clinical AR scenarios.

Abstract

Recovering a dense 3D body mesh from monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in surgical augmented reality (AR), where an anesthetized patient lies under surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic surgical sequences and three public HMR video benchmarks. Under surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.