Layout-induced Video Representation for Recognizing Agent-in-Place Actions

TL;DR

This paper proposes a novel scene layout-based video representation, LIVR, that improves recognition of agent-in-place actions in outdoor surveillance by capturing geometric relationships and generalizing across unseen layouts.

Contribution

The introduction of LIVR, a layout-induced video representation that encodes scene geometry and topology, enabling better generalization to unseen outdoor scene layouts for agent-in-place action recognition.

Findings

LIVR improves generalization to unseen layouts.

The Agent-in-Place Action dataset supports the effectiveness of LIVR.

LIVR outperforms baseline methods in recognition accuracy.

Abstract

We address the recognition of agent-in-place actions, which are associated with agents who perform them and places where they occur, in the context of outdoor home surveillance. We introduce a representation of the geometry and topology of scene layouts so that a network can generalize from the layouts observed in the training set to unseen layouts in the test set. This Layout-Induced Video Representation (LIVR) abstracts away low-level appearance variance and encodes geometric and topological relationships of places in a specific scene layout. LIVR partitions the semantic features of a video clip into different places to force the network to learn place-based feature descriptions; to predict the confidence of each action, LIVR aggregates features from the place associated with an action and its adjacent places on the scene layout. We introduce the Agent-in-Place Action dataset to show that our method allows neural network models to generalize significantly better to unseen scenes.