Representation learning from videos in-the-wild: An object-centric approach

TL;DR

This paper introduces an object-centric representation learning method from uncurated videos, combining supervised object detection and self-supervised learning to improve transfer learning and out-of-distribution generalization.

Contribution

It presents a novel approach that integrates object detection with self-supervised learning for video-based representation learning, demonstrating broad improvements across multiple benchmarks.

Findings

Improves performance on 18/19 few-shot learning tasks

Enhances results on all 8 out-of-distribution generalization tasks

Shows benefits of combining supervised and self-supervised losses in video learning

Abstract

We propose a method to learn image representations from uncurated videos. We combine a supervised loss from off-the-shelf object detectors and self-supervised losses which naturally arise from the video-shot-frame-object hierarchy present in each video. We report competitive results on 19 transfer learning tasks of the Visual Task Adaptation Benchmark (VTAB), and on 8 out-of-distribution-generalization tasks, and discuss the benefits and shortcomings of the proposed approach. In particular, it improves over the baseline on all 18/19 few-shot learning tasks and 8/8 out-of-distribution generalization tasks. Finally, we perform several ablation studies and analyze the impact of the pretrained object detector on the performance across this suite of tasks.