DesCo: Learning Object Recognition with Rich Language Descriptions

TL;DR

This paper introduces DesCo, a novel approach that enhances object recognition by using rich language descriptions generated by large language models and context-sensitive queries, significantly improving zero-shot detection performance.

Contribution

The paper proposes a new description-conditioned learning paradigm that leverages large language models and context-aware queries to improve visual recognition with detailed language descriptions.

Findings

Achieves state-of-the-art zero-shot detection results on LVIS and OminiLabel benchmarks.

Outperforms previous models GLIP and FIBER by large margins.

Demonstrates the effectiveness of rich language descriptions and context-sensitive queries in object recognition.

Abstract

Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. "a photo of a cat") and improve the models' adaptability to identify novel objects and domains. Recently, several studies have attempted to query these models with complex language expressions that include specifications of fine-grained semantic details, such as attributes, shapes, textures, and relations. However, simply incorporating language descriptions as queries does not guarantee accurate interpretation by the models. In fact, our experiments show that GLIP, the state-of-the-art vision-language model for object detection, often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenges, we propose a new description-conditioned (DesCo) paradigm of learning object recognition models with rich language descriptions consisting of two major innovations: 1) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects based on object names and the raw image-text caption; 2) we design context-sensitive queries to improve the model's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models, GLIP and FIBER, by a large margin.