Action Classification and Highlighting in Videos

TL;DR

This paper introduces an attention-based LSTM model for human activity recognition in videos, which jointly classifies actions and highlights relevant frames by attending to salient visual features, outperforming existing methods.

Contribution

The paper proposes a novel attention-based LSTM model that jointly learns to classify actions and highlight key frames, leveraging semantic visual features for improved accuracy.

Findings

Outperforms vanilla LSTM and CNN models in action classification.

Effectively attends to important objects and scene information.

Achieves superior results on the ActivityNet dataset.

Abstract

Inspired by recent advances in neural machine translation, that jointly align and translate using encoder-decoder networks equipped with attention, we propose an attentionbased LSTM model for human activity recognition. Our model jointly learns to classify actions and highlight frames associated with the action, by attending to salient visual information through a jointly learned soft-attention networks. We explore attention informed by various forms of visual semantic features, including those encoding actions, objects and scenes. We qualitatively show that soft-attention can learn to effectively attend to important objects and scene information correlated with specific human actions. Further, we show that, quantitatively, our attention-based LSTM outperforms the vanilla LSTM and CNN models used by stateof-the-art methods. On a large-scale youtube video dataset, ActivityNet, our model outperforms competing methods in action classification.