To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression

TL;DR

This paper introduces an attention-based approach for localizing specific sentence descriptions within untrimmed videos by leveraging global context and multi-modal co-attention mechanisms, improving accuracy and efficiency.

Contribution

The paper presents a novel Attention Based Location Regression (ABLR) method that encodes video and sentence context, employs co-attention for crucial detail highlighting, and predicts temporal boundaries end-to-end.

Findings

Outperforms existing methods on ActivityNet Captions and TACoS datasets.

Effectively captures global video structure and sentence details.

Demonstrates improved localization accuracy and computational efficiency.

Abstract

Given an untrimmed video and a sentence description, temporal sentence localization aims to automatically determine the start and end points of the described sentence within the video. The problem is challenging as it needs the understanding of both video and sentence. Existing research predominantly employs a costly "scan and localize" framework, neglecting the global video context and the specific details within sentences which play as critical issues for this problem. In this paper, we propose a novel Attention Based Location Regression (ABLR) approach to solve the temporal sentence localization from a global perspective. Specifically, to preserve the context information, ABLR first encodes both video and sentence via Bidirectional LSTM networks. Then, a multi-modal co-attention mechanism is introduced to generate not only video attention which reflects the global video structure, but also sentence attention which highlights the crucial details for temporal localization. Finally, a novel attention based location regression network is designed to predict the temporal coordinates of sentence query from the previous attention. ABLR is jointly trained in an end-to-end manner. Comprehensive experiments on ActivityNet Captions and TACoS datasets demonstrate both the effectiveness and the efficiency of the proposed ABLR approach.