Temporal Action Localization using Long Short-Term Dependency

TL;DR

This paper introduces the Gemini Network, a novel approach for temporal action localization in videos that leverages dual subnets, parallel feature pipelines, and auxiliary supervision to improve modeling of temporal structures and achieve state-of-the-art results.

Contribution

The paper presents a new Gemini Network architecture that enhances temporal structure modeling through dual subnets, parallel feature extraction, and auxiliary supervision, outperforming existing methods.

Findings

Achieved state-of-the-art performance on THUMOS14 dataset.

Outperformed previous methods on ActivityNet dataset.

Demonstrated effective modeling of temporal structures in untrimmed videos.

Abstract

Temporal action localization in untrimmed videos is an important but difficult task. Difficulties are encountered in the application of existing methods when modeling temporal structures of videos. In the present study, we developed a novel method, referred to as Gemini Network, for effective modeling of temporal structures and achieving high-performance temporal action localization. The significant improvements afforded by the proposed method are attributable to three major factors. First, the developed network utilizes two subnets for effective modeling of temporal structures. Second, three parallel feature extraction pipelines are used to prevent interference between the extractions of different stage features. Third, the proposed method utilizes auxiliary supervision, with the auxiliary classifier losses affording additional constraints for improving the modeling capability of the network. As a demonstration of its effectiveness, the Gemini Network was used to achieve state-of-the-art temporal action localization performance on two challenging datasets, namely, THUMOS14 and ActivityNet.