Skeletal Data Matching and Merging from Multiple RGB-D Sensors for Room-Scale Human Behaviour Tracking

TL;DR

This paper presents a method for fusing data from multiple RGB-D sensors to improve room-scale human behavior tracking, addressing calibration, skeleton matching, and merging challenges to enable occlusion-resilient tracking.

Contribution

It introduces a novel approach to align point clouds and merge skeleton data from multiple sensors, enhancing accuracy and robustness in human behavior tracking.

Findings

Successful alignment of point clouds from multiple sensors.

Effective skeleton matching and merging across sensors.

Improved occlusion resilience in human tracking.

Abstract

A popular and affordable option to provide room-scale human behaviour tracking is to rely on commodity RGB-D sensors %todo: such as the Kinect family of devices? as such devices offer body tracking capabilities at a reasonable price point. While their capabilities may be sufficient for applications such as entertainment systems where a person plays in front of a television, RGB-D sensors are sensitive to occlusions from objects or other persons that might be in the way in more complex room-scale setups. To alleviate the occlusion issue but also in order to extend the tracking range and strengthen its accuracy, it is possible to rely on multiple RGB-D sensors and perform data fusion. Unfortunately, fusing the data in a meaningful manner raises additional challenges related to the calibration of the sensors relative to each other to provide a common frame of reference, but also regarding skeleton matching and merging when actually combining the data. In this paper, we discuss our approach to tackle these challenges and present the results we achieved, through aligned point clouds and combined skeleton lists. These results successfully enable unobtrusive and occlusion-resilient human behaviour tracking at room scale, that may be used as input for interactive applications as well as (possibly remote) collaborative systems.