CaSAR: Contact-aware Skeletal Action Recognition

TL;DR

CaSAR introduces a contact-aware skeletal action recognition framework that captures hand-object interactions through novel spatial representations, significantly improving accuracy in egocentric action recognition tasks.

Contribution

The paper proposes a new contact-aware approach that models hand-object interactions using contact points and distant points, enhancing recognition performance.

Findings

Achieves state-of-the-art accuracy of 91.3% on H2O dataset.

Achieves state-of-the-art accuracy of 98.4% on FPHA dataset.

Effectively models hand-object interactions for egocentric action recognition.

Abstract

Skeletal Action recognition from an egocentric view is important for applications such as interfaces in AR/VR glasses and human-robot interaction, where the device has limited resources. Most of the existing skeletal action recognition approaches use 3D coordinates of hand joints and 8-corner rectangular bounding boxes of objects as inputs, but they do not capture how the hands and objects interact with each other within the spatial context. In this paper, we present a new framework called Contact-aware Skeletal Action Recognition (CaSAR). It uses novel representations of hand-object interaction that encompass spatial information: 1) contact points where the hand joints meet the objects, 2) distant points where the hand joints are far away from the object and nearly not involved in the current action. Our framework is able to learn how the hands touch or stay away from the objects for each frame of the action sequence, and use this information to predict the action class. We demonstrate that our approach achieves the state-of-the-art accuracy of 91.3% and 98.4% on two public datasets, H2O and FPHA, respectively.