Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency

TL;DR

This paper introduces HiCo, a hierarchical consistency learning framework that leverages untrimmed videos by capturing visual and topical consistencies, leading to improved video representations over traditional methods.

Contribution

The paper proposes a novel hierarchical consistency learning framework, HiCo, that effectively utilizes untrimmed videos for self-supervised representation learning, surpassing existing trimmed-video-based approaches.

Findings

HiCo produces stronger video representations from untrimmed videos.

It improves representation quality when applied to trimmed videos.

Hierarchical consistency learning outperforms standard contrastive methods.

Abstract

Natural videos provide rich visual contents for self-supervised learning. Yet most existing approaches for learning spatio-temporal representations rely on manually trimmed videos, leading to limited diversity in visual patterns and limited performance gain. In this work, we aim to learn representations by leveraging more abundant information in untrimmed videos. To this end, we propose to learn a hierarchy of consistencies in videos, i.e., visual consistency and topical consistency, corresponding respectively to clip pairs that tend to be visually similar when separated by a short time span and share similar topics when separated by a long time span. Specifically, a hierarchical consistency learning framework HiCo is presented, where the visually consistent pairs are encouraged to have the same representation through contrastive learning, while the topically consistent pairs are coupled through a topical classifier that distinguishes whether they are topic related. Further, we impose a gradual sampling algorithm for proposed hierarchical consistency learning, and demonstrate its theoretical superiority. Empirically, we show that not only HiCo can generate stronger representations on untrimmed videos, it also improves the representation quality when applied to trimmed videos. This is in contrast to standard contrastive learning that fails to learn appropriate representations from untrimmed videos.