Unsupervised Open-Vocabulary Object Localization in Videos

TL;DR

This paper introduces an unsupervised method for localizing objects in videos by leveraging pre-trained vision-language models and slot attention, achieving strong results without manual annotations.

Contribution

It presents a novel unsupervised approach combining slot attention and CLIP for video object localization, eliminating the need for manual labels.

Findings

Effective localization on standard video benchmarks

Utilizes pre-trained CLIP model for semantic understanding

First unsupervised method to achieve competitive results

Abstract

In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization. We propose a method that first localizes objects in videos via an object-centric approach with slot attention and then assigns text to the obtained slots. The latter is achieved by an unsupervised way to read localized semantic information from the pre-trained CLIP model. The resulting video object localization is entirely unsupervised apart from the implicit annotation contained in CLIP, and it is effectively the first unsupervised approach that yields good results on regular video benchmarks.