Multimodal Visual-haptic pose estimation in the presence of transient occlusion

TL;DR

This paper presents a multimodal approach combining vision and capacitive sensing with a modified observer model to improve human pose estimation during transient occlusion in human-robot collaboration.

Contribution

It introduces a novel combination of deep predictive coding vision and capacitive sensing with a modified Luenberger observer for robust pose estimation.

Findings

Outperforms single-sensor methods in occlusion scenarios

Effective in estimating human forearm pose during transient occlusion

Demonstrates improved safety in human-robot collaboration environments

Abstract

Human-robot collaboration requires the establishment of methods to guarantee the safety of participating operators. A necessary part of this process is ensuring reliable human pose estimation. Established vision-based modalities encounter problems when under conditions of occlusion. This article describes the combination of two perception modalities for pose estimation in environments containing such transient occlusion. We first introduce a vision-based pose estimation method, based on a deep Predictive Coding (PC) model featuring robustness to partial occlusion. Next, capacitive sensing hardware capable of detecting various objects is introduced. The sensor is compact enough to be mounted on the exterior of any given robotic system. The technology is particularly well-suited to detection of capacitive material, such as living tissue. Pose estimation from the two individual sensing modalities is combined using a modified Luenberger observer model. We demonstrate that the results offer better performance than either sensor alone. The efficacy of the system is demonstrated on an environment containing a robot arm and a human, showing the ability to estimate the pose of a human forearm under varying levels of occlusion.