Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo

TL;DR

This paper introduces a novel stereo vision-based method for real-time 3D hand tracking that bypasses depth reconstruction, achieving comparable or superior accuracy to RGBD-based approaches in various interaction scenarios.

Contribution

It presents a new optimization-based approach that uses color consistency for 3D hand tracking from stereo images, eliminating the need for dense 3D reconstruction.

Findings

Stereo-based method matches RGBD accuracy in standard datasets

The approach performs well in real-time hand and object interaction scenarios

It can outperform RGBD methods in certain cases

Abstract

We present a novel solution to the problem of 3D tracking of the articulated motion of human hand(s), possibly in interaction with other objects. The vast majority of contemporary relevant work capitalizes on depth information provided by RGBD cameras. In this work, we show that accurate and efficient 3D hand tracking is possible, even for the case of RGB stereo. A straightforward approach for solving the problem based on such input would be to first recover depth and then apply a state of the art depth-based 3D hand tracking method. Unfortunately, this does not work well in practice because the stereo-based, dense 3D reconstruction of hands is far less accurate than the one obtained by RGBD cameras. Our approach bypasses 3D reconstruction and follows a completely different route: 3D hand tracking is formulated as an optimization problem whose solution is the hand configuration that maximizes the color consistency between the two views of the hand. We demonstrate the applicability of our method for real time tracking of a single hand, of a hand manipulating an object and of two interacting hands. The method has been evaluated quantitatively on standard datasets and in comparison to relevant, state of the art RGBD-based approaches. The obtained results demonstrate that the proposed stereo-based method performs equally well to its RGBD-based competitors, and in some cases, it even outperforms them.