Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data

TL;DR

This paper advances activity recognition by focusing on fine-grained and composite activities, introducing a new dataset, and demonstrating the effectiveness of hand-centric features and script data for improved detection and recognition.

Contribution

It presents a new dataset with detailed annotations, compares hand-centric features with holistic ones, and leverages script data for recognizing novel composite activities.

Findings

Hand-centric features outperform holistic features for fine-grained activity detection.

Decomposing composite activities into attributes enables better recognition and transfer.

Using script data allows recognition of unseen composite activities without training examples.

Abstract

Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts. We show the benefits of our hand-centric approach for fine-grained activity classification and detection. For composite activity recognition we find that decomposition into attributes allows sharing information across composites and is essential to attack this hard task. Using script data we can recognize novel composites without having training data for them.