Bridging the gap to real-world language-grounded visual concept learning

TL;DR

This paper introduces a scalable, adaptive framework for language-grounded visual concept learning in real-world scenes, enabling diverse concept axes discovery and manipulation without prior knowledge or additional parameters.

Contribution

It proposes a universal prompting strategy and compositional anchoring objective for adaptive concept axis identification and grounding in real-world images.

Findings

Outperforms existing methods in visual concept editing tasks.

Demonstrates strong compositional generalization across diverse datasets.

Effectively discovers and manipulates multiple concept axes without prior knowledge.

Abstract

Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. However, existing approaches to language-grounded visual concept learning are limited to a few predefined primitive axes, such as color and shape, and are typically explored in synthetic datasets. In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Leveraging a pretrained vision-language model and our universal prompting strategy, our framework identifies a diverse image-related axes without any prior knowledge. Our universal concept encoder adaptively binds visual features to the discovered axes without introducing additional model parameters for each concept. To ground visual concepts along the discovered axes, we optimize a compositional anchoring objective, which ensures that each axis can be independently manipulated without affecting others. We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ, and AFHQ, showcasing superior editing capabilities across diverse real-world concepts that are too varied to be manually predefined. Our method also exhibits strong compositional generalization, outperforming existing visual concept learning and text-based editing methods. The code is available at https://github.com/whieya/Language-grounded-VCL.