Multi-View Active Sensing for Human-Robot Interaction via Hierarchically Connected Tree

TL;DR

This paper introduces a multi-camera active sensing system with a hierarchically connected tree structure to improve human perception in human-robot interaction, significantly enhancing keypart recognition and obstacle avoidance.

Contribution

The work presents a novel hierarchical tree-based fusion strategy for multi-source RGB-D data, improving perception accuracy and robustness in dynamic human-robot environments.

Findings

Keypart recognition recall improved from 69.20% to 90.10%.

Enhanced obstacle avoidance capabilities of the robotic arm.

Effective handling of occlusion and localization challenges.

Abstract

Comprehensive perception of human beings is the prerequisite to ensure the safety of human-robot interaction. Currently, prevailing visual sensing approach typically involves a single static camera, resulting in a restricted and occluded field of view. In our work, we develop an active vision system using multiple cameras to dynamically capture multi-source RGB-D data. An integrated human sensing strategy based on a hierarchically connected tree structure is proposed to fuse localized visual information. Constituting the tree model are the nodes representing keypoints and the edges representing keyparts, which are consistently interconnected to preserve the structural constraints during multi-source fusion. Utilizing RGB-D data and HRNet, the 3D positions of keypoints are analytically estimated, and their presence is inferred through a sliding widow of confidence scores. Subsequently, the point clouds of reliable keyparts are extracted by drawing occlusion-resistant masks, enabling fine registration between data clouds and cylindrical model following the hierarchical order. Experimental results demonstrate that our method enhances keypart recognition recall from 69.20% to 90.10%, compared to employing a single static camera. Furthermore, in overcoming challenges related to localized and occluded perception, the robotic arm's obstacle avoidance capabilities are effectively improved.