Action Recognition based on Cross-Situational Action-object Statistics

TL;DR

This paper investigates how training data properties influence the generalization of visual action recognition models, inspired by cross-situational learning, and identifies key data features that improve classifier robustness.

Contribution

It introduces a data-centric approach inspired by cognitive mechanisms to enhance action recognition models' generalization by identifying crucial training data properties.

Findings

Certain action-object co-occurrence properties improve generalization.

Typical datasets lack these key properties, limiting model robustness.

Guidelines for constructing better training datasets are proposed.

Abstract

Machine learning models of visual action recognition are typically trained and tested on data from specific situations where actions are associated with certain objects. It is an open question how action-object associations in the training set influence a model's ability to generalize beyond trained situations. We set out to identify properties of training data that lead to action recognition models with greater generalization ability. To do this, we take inspiration from a cognitive mechanism called cross-situational learning, which states that human learners extract the meaning of concepts by observing instances of the same concept across different situations. We perform controlled experiments with various types of action-object associations, and identify key properties of action-object co-occurrence in training data that lead to better classifiers. Given that these properties are missing in the datasets that are typically used to train action classifiers in the computer vision literature, our work provides useful insights on how we should best construct datasets for efficiently training for better generalization.