Hierarchical Video Understanding

TL;DR

This paper presents a hierarchical video understanding model that captures actions at multiple levels of detail, improving performance by leveraging the structure of real-world activities.

Contribution

It introduces a hierarchical architecture trained with a joint loss to learn coarse and fine-grained video targets simultaneously, enhancing understanding accuracy.

Findings

Models exploiting multiple granularity levels outperform single-level models.

Hierarchical training improves accuracy on coarse, fine, and captioning tasks.

Empirical validation on the Something-Something dataset demonstrates effectiveness.

Abstract

We introduce a hierarchical architecture for video understanding that exploits the structure of real world actions by capturing targets at different levels of granularity. We design the model such that it first learns simpler coarse-grained tasks, and then moves on to learn more fine-grained targets. The model is trained with a joint loss on different granularity levels. We demonstrate empirical results on the recent release of Something-Something dataset, which provides a hierarchy of targets, namely coarse-grained action groups, fine-grained action categories, and captions. Experiments suggest that models that exploit targets at different levels of granularity achieve better performance on all levels.