Learning to Name Classes for Vision and Language Models

TL;DR

This paper introduces a method to learn optimal class-specific word embeddings from visual data, improving zero-shot recognition, dataset adaptation, and handling ambiguous class names in vision-language models.

Contribution

It proposes a novel approach to adapt class names by learning word embeddings from visual content, enhancing model flexibility and performance.

Findings

Significant performance improvements in image classification and object detection

Effective adaptation to new datasets with minimal fine-tuning

Ability to correct or refine class names based on learned embeddings

Abstract

Large scale vision and language models can achieve impressive zero-shot recognition performance by mapping class specific text queries to image content. Two distinct challenges that remain however, are high sensitivity to the choice of handcrafted class names that define queries, and the difficulty of adaptation to new, smaller datasets. Towards addressing these problems, we propose to leverage available data to learn, for each class, an optimal word embedding as a function of the visual content. By learning new word embeddings on an otherwise frozen model, we are able to retain zero-shot capabilities for new classes, easily adapt models to new datasets, and adjust potentially erroneous, non-descriptive or ambiguous class names. We show that our solution can easily be integrated in image classification and object detection pipelines, yields significant performance gains in multiple scenarios and provides insights into model biases and labelling errors.