Exploring Temporal Preservation Networks for Precise Temporal Action Localization

TL;DR

This paper introduces the Temporal Preservation Convolutional (TPC) Network, which enhances 3D ConvNets with temporal resolution preservation, enabling more precise frame-level action localization in videos.

Contribution

The TPC network fully preserves temporal resolution during processing, allowing for more accurate per-frame action predictions compared to previous methods.

Findings

Significant improvement in per-frame action prediction accuracy

Competitive results on segment-level localization

End-to-end trainable architecture

Abstract

Temporal action localization is an important task of computer vision. Though a variety of methods have been proposed, it still remains an open question how to predict the temporal boundaries of action segments precisely. Most works use segment-level classifiers to select video segments pre-determined by action proposal or dense sliding windows. However, in order to achieve more precise action boundaries, a temporal localization system should make dense predictions at a fine granularity. A newly proposed work exploits Convolutional-Deconvolutional-Convolutional (CDC) filters to upsample the predictions of 3D ConvNets, making it possible to perform per-frame action predictions and achieving promising performance in terms of temporal action localization. However, CDC network loses temporal information partially due to the temporal downsampling operation. In this paper, we propose an elegant and powerful Temporal Preservation Convolutional (TPC) Network that equips 3D ConvNets with TPC filters. TPC network can fully preserve temporal resolution and downsample the spatial resolution simultaneously, enabling frame-level granularity action localization. TPC network can be trained in an end-to-end manner. Experiment results on public datasets show that TPC network achieves significant improvement on per-frame action prediction and competing results on segment-level temporal action localization.