Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wild

TL;DR

Chameleon is a versatile, data-efficient model capable of adapting to unseen dense visual prediction tasks in various real-world scenarios using minimal labeled data, advancing the field of vision generalists.

Contribution

This work introduces a universal meta-learning-based model that adapts to diverse dense visual tasks with few examples, addressing a key challenge in low-data regimes.

Findings

Successfully adapts to video, 3D, medical, biological, and user-interactive tasks.

Operates effectively with at most 50 labeled images per task.

Outperforms existing data-efficient vision generalists.

Abstract

Large language models have evolved data-efficient generalists, benefiting from the universal language interface and large-scale pre-training. However, constructing a data-efficient generalist for dense visual prediction presents a distinct challenge due to the variation in label structures across different tasks. Consequently, generalization to unseen dense prediction tasks in the low-data regime is not straightforward and has received less attention from previous vision generalists. In this study, we explore a universal model that can flexibly adapt to unseen dense label structures with a few examples, enabling it to serve as a data-efficient vision generalist in diverse real-world scenarios. To this end, we base our method on a powerful meta-learning framework and explore several axes to improve its performance and versatility for real-world problems, such as flexible adaptation mechanisms and scalability. We evaluate our model across a spectrum of unseen real-world scenarios where low-shot learning is desirable, including video, 3D, medical, biological, and user-interactive tasks. Equipped with a generic architecture and an effective adaptation mechanism, our model flexibly adapts to all of these tasks with at most 50 labeled images, showcasing a significant advancement over existing data-efficient generalist approaches. Codes are available at https://github.com/GitGyun/chameleon.