Entity-aware and Motion-aware Transformers for Language-driven Action Localization in Videos

TL;DR

This paper introduces entity-aware and motion-aware Transformers that improve language-driven action localization in videos by coarsely locating clips and precisely predicting boundaries using motion cues.

Contribution

The paper presents a novel Transformer-based framework that integrates entity and motion information for more accurate action localization in videos.

Findings

Outperforms existing methods on Charades-STA and TACoS datasets.

Effectively combines entity and motion cues for better boundary prediction.

Enhances visual-linguistic alignment with cross-modal and cross-frame attention.

Abstract

Language-driven action localization in videos is a challenging task that involves not only visual-linguistic matching but also action boundary prediction. Recent progress has been achieved through aligning language query to video segments, but estimating precise boundaries is still under-explored. In this paper, we propose entity-aware and motion-aware Transformers that progressively localizes actions in videos by first coarsely locating clips with entity queries and then finely predicting exact boundaries in a shrunken temporal region with motion queries. The entity-aware Transformer incorporates the textual entities into visual representation learning via cross-modal and cross-frame attentions to facilitate attending action-related video clips. The motion-aware Transformer captures fine-grained motion changes at multiple temporal scales via integrating long short-term memory into the self-attention module to further improve the precision of action boundary prediction. Extensive experiments on the Charades-STA and TACoS datasets demonstrate that our method achieves better performance than existing methods.