Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition

TL;DR

This paper explores zero-shot classification by description using CLIP, addressing its limitations in recognizing detailed object attributes without class names, and introduces new datasets, methods, and architectural modifications to improve part-attribute detection.

Contribution

It introduces a new zero-shot classification task based on descriptions, a dataset for evaluation, and architectural enhancements to CLIP for better part-attribute recognition.

Findings

Improved CLIP performance on fine-grained attribute detection

New datasets for zero-shot description-based classification

Enhanced CLIP architecture with multi-resolution approach

Abstract

In this study, we define and tackle zero shot "real" classification by description, a novel task that evaluates the ability of Vision-Language Models (VLMs) like CLIP to classify objects based solely on descriptive attributes, excluding object class names. This approach highlights the current limitations of VLMs in understanding intricate object descriptions, pushing these models beyond mere object recognition. To facilitate this exploration, we introduce a new challenge and release description data for six popular fine-grained benchmarks, which omit object names to encourage genuine zero-shot learning within the research community. Additionally, we propose a method to enhance CLIP's attribute detection capabilities through targeted training using ImageNet21k's diverse object categories, paired with rich attribute descriptions generated by large language models. Furthermore, we introduce a modified CLIP architecture that leverages multiple resolutions to improve the detection of fine-grained part attributes. Through these efforts, we broaden the understanding of part-attribute recognition in CLIP, improving its performance in fine-grained classification tasks across six popular benchmarks, as well as in the PACO dataset, a widely used benchmark for object-attribute recognition. Code is available at: https://github.com/ethanbar11/grounding_ge_public.