A Universal Action Space for General Behavior Analysis

TL;DR

This paper introduces a Universal Action Space (UAS) built from large-scale labeled datasets, enabling improved analysis and categorization of animal and human behaviors using deep learning representations.

Contribution

The paper proposes a novel Universal Action Space derived from existing datasets, facilitating general behavior analysis across species with deep neural network features.

Findings

UAS enables effective behavior categorization across species

Deep learning features improve robustness in behavior analysis

Source code is publicly available for reproducibility

Abstract

Analyzing animal and human behavior has long been a challenging task in computer vision. Early approaches from the 1970s to the 1990s relied on hand-crafted edge detection, segmentation, and low-level features such as color, shape, and texture to locate objects and infer their identities-an inherently ill-posed problem. Behavior analysis in this era typically proceeded by tracking identified objects over time and modeling their trajectories using sparse feature points, which further limited robustness and generalization. A major shift occurred with the introduction of ImageNet by Deng and Li in 2010, which enabled large-scale visual recognition through deep neural networks and effectively served as a comprehensive visual dictionary. This development allowed object recognition to move beyond complex low-level processing toward learned high-level representations. In this work, we follow this paradigm to build a large-scale Universal Action Space (UAS) using existing labeled human-action datasets. We then use this UAS as the foundation for analyzing and categorizing mammalian and chimpanzee behavior datasets. The source code is released on GitHub at https://github.com/franktpmvu/Universal-Action-Space.