QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries

TL;DR

QueryAdapter is a fast, unsupervised framework that adapts vision-language models to natural language queries in robotic perception, improving object retrieval and generalization in real-world scenarios.

Contribution

It introduces a novel prompt-based adaptation method that uses unlabelled data and negative class labels for rapid, effective VLM adaptation to diverse natural language queries.

Findings

Significantly improves object retrieval performance over existing methods.

Adapts in minutes using unlabelled data during deployment.

Shows strong generalization to new datasets and query types.

Abstract

A domain shift exists between the large-scale, internet data used to train a Vision-Language Model (VLM) and the raw image streams collected by a robot. Existing adaptation strategies require the definition of a closed-set of classes, which is impractical for a robot that must respond to diverse natural language queries. In response, we present QueryAdapter; a novel framework for rapidly adapting a pre-trained VLM in response to a natural language query. QueryAdapter leverages unlabelled data collected during previous deployments to align VLM features with semantic classes related to the query. By optimising learnable prompt tokens and actively selecting objects for training, an adapted model can be produced in a matter of minutes. We also explore how objects unrelated to the query should be dealt with when using real-world data for adaptation. In turn, we propose the use of object captions as negative class labels, helping to produce better calibrated confidence scores during adaptation. Extensive experiments on ScanNet++ demonstrate that QueryAdapter significantly enhances object retrieval performance compared to state-of-the-art unsupervised VLM adapters and 3D scene graph methods. Furthermore, the approach exhibits robust generalization to abstract affordance queries and other datasets, such as Ego4D.