Localization-Aware Multi-Scale Representation Learning for Repetitive Action Counting

TL;DR

This paper presents a novel localization-aware multi-scale learning framework that improves repetitive action counting in videos by reducing noise impact and capturing flexible temporal correlations.

Contribution

It introduces a localization-aware multi-scale representation learning framework with scale-specific and localization modules for robust, noise-resistant action counting.

Findings

Outperforms existing methods on RepCountA and UCFRep datasets.

Effectively reduces noise impact in repetitive action counting.

Enhances temporal correlation modeling across various action frequencies.

Abstract

Repetitive action counting (RAC) aims to estimate the number of class-agnostic action occurrences in a video without exemplars. Most current RAC methods rely on a raw frame-to-frame similarity representation for period prediction. However, this approach can be significantly disrupted by common noise such as action interruptions and inconsistencies, leading to sub-optimal counting performance in realistic scenarios. In this paper, we introduce a foreground localization optimization objective into similarity representation learning to obtain more robust and efficient video features. We propose a Localization-Aware Multi-Scale Representation Learning (LMRL) framework. Specifically, we apply a Multi-Scale Period-Aware Representation (MPR) with a scale-specific design to accommodate various action frequencies and learn more flexible temporal correlations. Furthermore, we introduce the Repetition Foreground Localization (RFL) method, which enhances the representation by coarsely identifying periodic actions and incorporating global semantic information. These two modules can be jointly optimized, resulting in a more discerning periodic action representation. Our approach significantly reduces the impact of noise, thereby improving counting accuracy. Additionally, the framework is designed to be scalable and adaptable to different types of video content. Experimental results on the RepCountA and UCFRep datasets demonstrate that our proposed method effectively handles repetitive action counting.