Agent-Environment Network for Temporal Action Proposal Generation

TL;DR

This paper introduces a novel contextual Agent-Environment Network that models human agents and their interactions with the environment to improve temporal action proposal generation in videos, outperforming existing methods.

Contribution

It proposes a new agent-environment network architecture that captures local agent actions and global interactions, addressing limitations of previous attention-based models.

Findings

Outperforms state-of-the-art methods on THUMOS-14 and ActivityNet-1.3 datasets

Effective with different backbone networks like C3D and SlowFast

Demonstrates robustness across various action categories

Abstract

Temporal action proposal generation is an essential and challenging task that aims at localizing temporal intervals containing human actions in untrimmed videos. Most of existing approaches are unable to follow the human cognitive process of understanding the video context due to lack of attention mechanism to express the concept of an action or an agent who performs the action or the interaction between the agent and the environment. Based on the action definition that a human, known as an agent, interacts with the environment and performs an action that affects the environment, we propose a contextual Agent-Environment Network. Our proposed contextual AEN involves (i) agent pathway, operating at a local level to tell about which humans/agents are acting and (ii) environment pathway operating at a global level to tell about how the agents interact with the environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks, i.e C3D and SlowFast, show that our method robustly exhibits outperformance against state-of-the-art methods regardless of the employed backbone network.