Modeling long-term interactions to enhance action recognition

TL;DR

This paper introduces a hierarchical LSTM-based method that leverages object interaction semantics at frame and temporal levels to improve egocentric action recognition, outperforming existing methods without using motion cues.

Contribution

It presents a novel hierarchical LSTM architecture combined with a region-based CNN for enhanced understanding of actions in egocentric videos, emphasizing object interaction semantics.

Findings

Outperforms state-of-the-art on standard benchmarks

Both frame-level and temporal-level HLSTM contribute to accuracy

Does not rely on motion information for recognition

Abstract

In this paper, we propose a new approach to under-stand actions in egocentric videos that exploits the semantics of object interactions at both frame and temporal levels. At the frame level, we use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects and calculates the action score through a CNN formulation. This information is then fed to a Hierarchical LongShort-Term Memory Network (HLSTM) that captures temporal dependencies between actions within and across shots. Ablation studies thoroughly validate the proposed approach, showing in particular that both levels of the HLSTM architecture contribute to performance improvement. Furthermore, quantitative comparisons show that the proposed approach outperforms the state-of-the-art in terms of action recognition on standard benchmarks,without relying on motion information