HomE: Homography-Equivariant Video Representation Learning

TL;DR

HomE introduces a self-supervised video representation learning method that explicitly models homography equivariance, improving performance on action recognition and pedestrian intent prediction tasks.

Contribution

The paper proposes a novel homography-equivariant representation learning approach for multi-view videos, leveraging geometric relationships for better self-supervised learning.

Findings

Achieves 96.4% accuracy on UCF101 action classification.

Outperforms state-of-the-art by 6% on pedestrian intent prediction.

Obtains 91.2% accuracy for pedestrian action classification.

Abstract

Recent advances in self-supervised representation learning have enabled more efficient and robust model performance without relying on extensive labeled data. However, most works are still focused on images, with few working on videos and even fewer on multi-view videos, where more powerful inductive biases can be leveraged for self-supervision. In this work, we propose a novel method for representation learning of multi-view videos, where we explicitly model the representation space to maintain Homography Equivariance (HomE). Our method learns an implicit mapping between different views, culminating in a representation space that maintains the homography relationship between neighboring views. We evaluate our HomE representation via action recognition and pedestrian intent prediction as downstream tasks. On action classification, our method obtains 96.4% 3-fold accuracy on the UCF101 dataset, better than most state-of-the-art self-supervised learning methods. Similarly, on the STIP dataset, we outperform the state-of-the-art by 6% for pedestrian intent prediction one second into the future while also obtaining an accuracy of 91.2% for pedestrian action (cross vs. not-cross) classification. Code is available at https://github.com/anirudhs123/HomE.