MultiCounter: Multiple Action Agnostic Repetition Counting in Untrimmed Videos

TL;DR

MultiCounter is an end-to-end deep learning framework that accurately detects, tracks, and counts multiple repetitive actions in untrimmed videos, outperforming existing methods and operating in real-time.

Contribution

It introduces novel modules for context correlation and action-agnostic counting, setting new benchmarks in multi-instance repetitive action counting.

Findings

Significantly improves Period-mAP by 41.0%.

Reduces AvgMAE by 58.6%.

Operates in real-time on standard GPU hardware.

Abstract

Multi-instance Repetitive Action Counting (MRAC) aims to estimate the number of repetitive actions performed by multiple instances in untrimmed videos, commonly found in human-centric domains like sports and exercise. In this paper, we propose MultiCounter, a fully end-to-end deep learning framework that enables simultaneous detection, tracking, and counting of repetitive actions of multiple human instances. Specifically, MultiCounter incorporates two novel modules: 1) mixed spatiotemporal interaction for efficient context correlation across consecutive frames, and 2) task-specific heads for accurate perception of periodic boundaries and generalization for action-agnostic human instances. We train MultiCounter on a synthetic dataset called MultiRep generated from annotated real-world videos. Experiments on the MultiRep dataset validate the fundamental challenge of MRAC tasks and showcase the superiority of our proposed model. Compared to ByteTrack+RepNet, a solution that combines an advanced tracker with a single repetition counter, MultiCounter substantially improves Period-mAP by 41.0%, reduces AvgMAE by 58.6%, and increases AvgOBO 1.48 times. This sets a new benchmark in the field of MRAC. Moreover, MultiCounter runs in real-time on a commodity GPU server and is insensitive to the number of human instances in a video.