Learning the Predictability of the Future

TL;DR

This paper presents a novel framework that learns hierarchical, predictable features from unlabeled video data using hyperbolic geometry, enabling automatic abstraction level selection for improved future action prediction.

Contribution

It introduces a hyperbolic space-based predictive model that learns which features are predictable and at what abstraction level, without requiring labeled data.

Findings

Hierarchical representations improve action prediction accuracy.

Unlabeled video data can be used to learn meaningful action hierarchies.

Hyperbolic geometry effectively encodes hierarchical structure in the model.

Abstract

We introduce a framework for learning from unlabeled video what is predictable in the future. Instead of committing up front to features to predict, our approach learns from data which features are predictable. Based on the observation that hyperbolic geometry naturally and compactly encodes hierarchical structure, we propose a predictive model in hyperbolic space. When the model is most confident, it will predict at a concrete level of the hierarchy, but when the model is not confident, it learns to automatically select a higher level of abstraction. Experiments on two established datasets show the key role of hierarchical representations for action prediction. Although our representation is trained with unlabeled video, visualizations show that action hierarchies emerge in the representation.