Multi-granularity Generator for Temporal Action Proposal

TL;DR

This paper introduces a multi-granularity generator that combines coarse segment proposals and fine frame actionness evaluation to improve temporal action proposal accuracy in untrimmed videos, achieving state-of-the-art results.

Contribution

The paper presents a novel multi-granularity generator (MGG) that integrates segment proposal and frame actionness components for better temporal action localization.

Findings

Outperforms state-of-the-art on THUMOS-14 and ActivityNet-1.3 datasets.

End-to-end trainable model with superior proposal quality.

Improves video detection accuracy with proposal classification.

Abstract

Temporal action proposal generation is an important task, aiming to localize the video segments containing human actions in an untrimmed video. In this paper, we propose a multi-granularity generator (MGG) to perform the temporal action proposal from different granularity perspectives, relying on the video visual features equipped with the position embedding information. First, we propose to use a bilinear matching model to exploit the rich local information within the video sequence. Afterwards, two components, namely segment proposal producer (SPP) and frame actionness producer (FAP), are combined to perform the task of temporal action proposal at two distinct granularities. SPP considers the whole video in the form of feature pyramid and generates segment proposals from one coarse perspective, while FAP carries out a finer actionness evaluation for each video frame. Our proposed MGG can be trained in an end-to-end fashion. By temporally adjusting the segment proposals with fine-grained frame actionness information, MGG achieves the superior performance over state-of-the-art methods on the public THUMOS-14 and ActivityNet-1.3 datasets. Moreover, we employ existing action classifiers to perform the classification of the proposals generated by MGG, leading to significant improvements compared against the competing methods for the video detection task.