Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos

TL;DR

This paper introduces Memory Storyboard, a novel approach for streaming self-supervised learning from egocentric videos, utilizing temporal segmentation inspired by human memory to improve representation learning.

Contribution

It proposes a two-tier memory hierarchy and a storyboard-based temporal segmentation method for more effective learning from real-world video streams.

Findings

Outperforms state-of-the-art unsupervised continual learning methods on egocentric datasets.

Semantic representations learned are more meaningful and robust.

Memory Storyboard effectively summarizes long-term visual streams.

Abstract

Self-supervised learning holds the promise of learning good representations from real-world continuous uncurated data streams. However, most existing works in visual self-supervised learning focus on static images or artificial data streams. Towards exploring a more realistic learning substrate, we investigate streaming self-supervised learning from long-form real-world egocentric video streams. Inspired by the event segmentation mechanism in human perception and memory, we propose "Memory Storyboard" that groups recent past frames into temporal segments for more effective summarization of the past visual streams for memory replay. To accommodate efficient temporal segmentation, we propose a two-tier memory hierarchy: the recent past is stored in a short-term memory, and the storyboard temporal segments are then transferred to a long-term memory. Experiments on real-world egocentric video datasets including SAYCam and KrishnaCam show that contrastive learning objectives on top of storyboard frames result in semantically meaningful representations that outperform those produced by state-of-the-art unsupervised continual learning methods.