Learning to Discriminate Information for Online Action Detection: Analysis and Application

TL;DR

This paper introduces novel recurrent units, IDU and IIU, that improve online action detection and anticipation by explicitly discriminating relevant information, leading to state-of-the-art performance on benchmark datasets.

Contribution

The paper proposes the IDU and IIU units that explicitly discriminate relevant information, enhancing the learning of discriminative features for online action detection and anticipation.

Findings

Outperforms state-of-the-art methods on TVSeries and THUMOS-14 datasets.

Demonstrates the effectiveness of IDU and IIU through ablation studies.

Achieves significant improvements in online action detection and anticipation.

Abstract

Online action detection, which aims to identify an ongoing action from a streaming video, is an important subject in real-world applications. For this task, previous methods use recurrent neural networks for modeling temporal relations in an input sequence. However, these methods overlook the fact that the input image sequence includes not only the action of interest but background and irrelevant actions. This would induce recurrent units to accumulate unnecessary information for encoding features on the action of interest. To overcome this problem, we propose a novel recurrent unit, named Information Discrimination Unit (IDU), which explicitly discriminates the information relevancy between an ongoing action and others to decide whether to accumulate the input information. This enables learning more discriminative representations for identifying an ongoing action. In this paper, we further present a new recurrent unit, called Information Integration Unit (IIU), for action anticipation. Our IIU exploits the outputs from IDU as pseudo action labels as well as RGB frames to learn enriched features of observed actions effectively. In experiments on TVSeries and THUMOS-14, the proposed methods outperform state-of-the-art methods by a significant margin in online action detection and action anticipation. Moreover, we demonstrate the effectiveness of the proposed units by conducting comprehensive ablation studies.