Parallel mesh reconstruction streams for pose estimation of interacting hands

TL;DR

This paper introduces MSMR-Net, a multi-stream 3D mesh reconstruction network with parallel decoding paths that effectively captures global and local features for hand pose estimation from a single RGB image, especially in complex interactions.

Contribution

The novel multi-stream decoder with parallel cross-resolution trajectories improves hand pose estimation accuracy in interacting scenarios with minimal additional parameters.

Findings

Outperforms existing algorithms on FreiHAND dataset

Effective in hand-hand and hand-object interaction scenarios

Close hand interactions are particularly challenging

Abstract

We present a new multi-stream 3D mesh reconstruction network (MSMR-Net) for hand pose estimation from a single RGB image. Our model consists of an image encoder followed by a mesh-convolution decoder composed of connected graph convolution layers. In contrast to previous models that form a single mesh decoding path, our decoder network incorporates multiple cross-resolution trajectories that are executed in parallel. Thus, global and local information are shared to form rich decoding representations at minor additional parameter cost compared to the single trajectory network. We demonstrate the effectiveness of our method in hand-hand and hand-object interaction scenarios at various levels of interaction. To evaluate the former scenario, we propose a method to generate RGB images of closely interacting hands. Moreoever, we suggest a metric to quantify the degree of interaction and show that close hand interactions are particularly challenging. Experimental results show that the MSMR-Net outperforms existing algorithms on the hand-object FreiHAND dataset as well as on our own hand-hand dataset.