Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type

TL;DR

This paper presents a multi-stage framework for egocentric hand action recognition that leverages hand surface geometry and grasp type analysis, avoiding the need for complex 3D object data.

Contribution

It introduces a novel approach combining mean curvature of hand surfaces and grasp type learning to improve hand action recognition accuracy.

Findings

Using hand grasp type improves recognition performance.

Mean curvature encoding enhances hand surface representation.

The method does not require 3D object pose annotations.

Abstract

We introduce a multi-stage framework that uses mean curvature on a hand surface and focuses on learning interaction between hand and object by analyzing hand grasp type for hand action recognition in egocentric videos. The proposed method does not require 3D information of objects including 6D object poses which are difficult to annotate for learning an object's behavior while it interacts with hands. Instead, the framework synthesizes the mean curvature of the hand mesh model to encode the hand surface geometry in 3D space. Additionally, our method learns the hand grasp type which is highly correlated with the hand action. From our experiment, we notice that using hand grasp type and mean curvature of hand increases the performance of the hand action recognition.