Towards In-context Scene Understanding

TL;DR

This paper introduces Hummingbird, a model that uses in-context learning with nearest neighbor retrieval for scene understanding tasks, achieving near-specialist performance without task-specific finetuning.

Contribution

The paper presents a novel pretraining protocol and a retrieval-based in-context learning approach for scene understanding, enabling flexible task execution without finetuning.

Findings

Hummingbird performs various scene understanding tasks without modification.

It approaches the performance of finetuned specialist models.

It enables efficient new task learning in interactive settings.

Abstract

In-context learning$\unicode{x2013}$the ability to configure a model's behavior with different prompts$\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\unicode{x2013}$leveraging attention within and across images$\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.