TALL: Temporal Activity Localization via Language Query

TL;DR

This paper introduces TALL, a method for localizing activities in untrimmed videos using natural language queries, addressing the limitations of fixed activity classifiers by enabling flexible, query-based localization.

Contribution

It proposes a novel Cross-modal Temporal Regression Localizer (CTRL) that jointly models text and video for accurate activity localization based on language queries.

Findings

CTRL outperforms previous methods on TaCoS and Charades-STA datasets.

The approach effectively aligns language queries with video segments.

New dataset Charades-STA with sentence annotations was created for evaluation.

Abstract

This paper focuses on temporal localization of actions in untrimmed videos. Existing methods typically train classifiers for a pre-defined list of actions and apply them in a sliding window fashion. However, activities in the wild consist of a wide combination of actors, actions and objects; it is difficult to design a proper activity list that meets users' needs. We propose to localize activities by natural language queries. Temporal Activity Localization via Language (TALL) is challenging as it requires: (1) suitable design of text and video representations to allow cross-modal matching of actions and language queries; (2) ability to locate actions accurately given features from sliding windows of limited granularity. We propose a novel Cross-modal Temporal Regression Localizer (CTRL) to jointly model text query and video clips, output alignment scores and action boundary regression results for candidate clips. For evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA. We also build complex sentence queries in Charades-STA for test. Experimental results show that CTRL outperforms previous methods significantly on both datasets.