Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks

TL;DR

This paper introduces a method to quantify static versus dynamic information biases in deep spatiotemporal models, revealing prevalent static bias and proposing techniques to mitigate it for improved video analysis.

Contribution

It presents a novel approach for measuring static and dynamic biases in models and applies it across multiple video tasks, offering insights and debiasing strategies.

Findings

Most models are biased toward static information.

Some datasets are biased toward static rather than dynamic cues.

Individual channels can be selectively biased toward static or dynamic information.

Abstract

There is limited understanding of the information captured by deep spatiotemporal models in their intermediate representations. For example, while evidence suggests that action recognition algorithms are heavily influenced by visual appearance in single frames, no quantitative methodology exists for evaluating such static bias in the latent representation compared to bias toward dynamics. We tackle this challenge by proposing an approach for quantifying the static and dynamic biases of any spatiotemporal model, and apply our approach to three tasks, action recognition, automatic video object segmentation (AVOS) and video instance segmentation (VIS). Our key findings are: (i) Most examined models are biased toward static information. (ii) Some datasets that are assumed to be biased toward dynamics are actually biased toward static information. (iii) Individual channels in an architecture can be biased toward static, dynamic or a combination of the two. (iv) Most models converge to their culminating biases in the first half of training. We then explore how these biases affect performance on dynamically biased datasets. For action recognition, we propose StaticDropout, a semantically guided dropout that debiases a model from static information toward dynamics. For AVOS, we design a better combination of fusion and cross connection layers compared with previous architectures.