Object Priors for Classifying and Localizing Unseen Actions

TL;DR

This paper introduces a novel approach for classifying and localizing unseen human actions in videos using only image-based object information, leveraging spatial and semantic priors to achieve state-of-the-art results.

Contribution

It proposes a new framework combining spatial and semantic object priors for unseen action classification and localization without video training data.

Findings

Spatial and semantic object priors improve unseen action localization.

Using multiple languages enhances semantic matching accuracy.

Achieves state-of-the-art results on five action datasets.

Abstract

This work strives for the classification and localization of human actions in videos, without the need for any labeled video training examples. Where existing work relies on transferring global attribute or object information from seen to unseen action videos, we seek to classify and spatio-temporally localize unseen actions in videos from image-based object information only. We propose three spatial object priors, which encode local person and object detectors along with their spatial relations. On top we introduce three semantic object priors, which extend semantic matching through word embeddings with three simple functions that tackle semantic ambiguity, object discrimination, and object naming. A video embedding combines the spatial and semantic object priors. It enables us to introduce a new video retrieval task that retrieves action tubes in video collections based on user-specified objects, spatial relations, and object size. Experimental evaluation on five action datasets shows the importance of spatial and semantic object priors for unseen actions. We find that persons and objects have preferred spatial relations that benefit unseen action localization, while using multiple languages and simple object filtering directly improves semantic matching, leading to state-of-the-art results for both unseen action classification and localization.