Tree of Attributes Prompt Learning for Vision-Language Models

TL;DR

This paper introduces Tree of Attributes Prompt learning (TAP), a method that leverages structured attribute hierarchies generated by LLMs to improve vision-language model adaptation for various classification tasks.

Contribution

TAP distills structured knowledge graphs from LLMs and incorporates explicit visual attribute learning, enhancing zero-shot and few-shot classification performance.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Improves zero-shot base-to-novel generalization.

Enhances cross-dataset transfer and few-shot classification.

Abstract

Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a "concept - attribute - description" structure for each category, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset transfer, as well as few-shot classification across 11 diverse datasets. Code is available at https://github.com/HHenryD/TAP.