Efficient Action Counting with Dynamic Queries

TL;DR

This paper introduces a linear-complexity, query-based method for counting repeated actions in videos, improving scalability and accuracy over previous correlation matrix-based approaches, especially for long, unseen, or fast actions.

Contribution

It proposes a dynamic action query update scheme and inter-query contrastive learning, enabling open-set, efficient, and accurate temporal repetition counting.

Findings

Outperforms state-of-the-art by 26.5% in OBO accuracy on RepCountA

Reduces computational burden by 94.1%

Excels in long videos, unseen actions, and various speeds

Abstract

Temporal repetition counting aims to quantify the repeated action cycles within a video. The majority of existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize repeated action cycles with linear computational complexity. Based on this representation, we further develop two key components to tackle the essential challenges of temporal repetition counting. Firstly, to facilitate open-set action counting, we propose the dynamic update scheme on action queries. Unlike static action queries, this approach dynamically embeds video features into action queries, offering a more flexible and generalizable representation. Secondly, to distinguish between actions of interest and background noise actions, we incorporate inter-query contrastive learning to regularize the video representations corresponding to different action queries. As a result, our method significantly outperforms previous works, particularly in terms of long video sequences, unseen actions, and actions at various speeds. On the challenging RepCountA benchmark, we outperform the state-of-the-art method TransRAC by 26.5% in OBO accuracy, with a 22.7% mean error decrease and 94.1% computational burden reduction. Code is available at https://github.com/lizishi/DeTRC.