Action Dubber: Timing Audible Actions via Inflectional Flow

TL;DR

This paper presents a new task called Audible Action Temporal Localization, focusing on identifying when and where audible movements occur in videos, using a novel inflectional flow approach without relying on audio input.

Contribution

It introduces $TA^{2}Net$, a new architecture that estimates inflectional flow for precise timing and localization of audible actions, along with a new dataset $Audible623$ for benchmarking.

Findings

$TA^{2}Net$ outperforms existing methods on $Audible623$

The approach generalizes well to sound source localization and counting tasks

The dataset enables focused research on audible action localization

Abstract

We introduce the task of Audible Action Temporal Localization, which aims to identify the spatio-temporal coordinates of audible movements. Unlike conventional tasks such as action recognition and temporal action localization, which broadly analyze video content, our task focuses on the distinct kinematic dynamics of audible actions. It is based on the premise that key actions are driven by inflectional movements; for example, collisions that produce sound often involve abrupt changes in motion. To capture this, we propose $TA^{2}Net$, a novel architecture that estimates inflectional flow using the second derivative of motion to determine collision timings without relying on audio input. $TA^{2}Net$ also integrates a self-supervised spatial localization strategy during training, combining contrastive learning with spatial analysis. This dual design improves temporal localization accuracy and simultaneously identifies sound sources within video frames. To support this task, we introduce a new benchmark dataset, $Audible623$, derived from Kinetics and UCF101 by removing non-essential vocalization subsets. Extensive experiments confirm the effectiveness of our approach on $Audible623$ and show strong generalizability to other domains, such as repetitive counting and sound source localization. Code and dataset are available at https://github.com/WenlongWan/Audible623.