Temporal Relevance Analysis for Video Action Models

TL;DR

This paper analyzes how CNN-based video action models capture temporal information, revealing they focus on local, not long-range dependencies, and that temporal relevance does not strongly correlate with model performance.

Contribution

It introduces a new method to quantify temporal relationships in action models and provides comprehensive analysis of factors affecting temporal modeling in video recognition.

Findings

Models mainly capture local temporal information.

No strong correlation between temporal relevance and performance.

Models tend to neglect long-range dependencies.

Abstract

In this paper, we provide a deep analysis of temporal modeling for action recognition, an important but underexplored problem in the literature. We first propose a new approach to quantify the temporal relationships between frames captured by CNN-based action models based on layer-wise relevance propagation. We then conduct comprehensive experiments and in-depth analysis to provide a better understanding of how temporal modeling is affected by various factors such as dataset, network architecture, and input frames. With this, we further study some important questions for action recognition that lead to interesting findings. Our analysis shows that there is no strong correlation between temporal relevance and model performance; and action models tend to capture local temporal information, but less long-range dependencies. Our codes and models will be publicly available.