Interpretable Zero-shot Learning with Infinite Class Concepts

TL;DR

This paper introduces InfZSL, a novel zero-shot learning framework that uses large language models to generate and select highly interpretable, transferable, and discriminative class concepts, improving recognition of unseen classes.

Contribution

It proposes a new framework that dynamically generates an unlimited number of class concepts with LLMs and employs an entropy-based scoring to select the most effective ones, addressing hallucination issues.

Findings

Significant improvements on three benchmark datasets.

Generation of highly interpretable, image-grounded concepts.

Effective filtering of concepts to enhance transferability and discriminability.

Abstract

Zero-shot learning (ZSL) aims to recognize unseen classes by aligning images with intermediate class semantics, like human-annotated concepts or class definitions. An emerging alternative leverages Large-scale Language Models (LLMs) to automatically generate class documents. However, these methods often face challenges with transparency in the classification process and may suffer from the notorious hallucination problem in LLMs, resulting in non-visual class semantics. This paper redefines class semantics in ZSL with a focus on transferability and discriminability, introducing a novel framework called Zero-shot Learning with Infinite Class Concepts (InfZSL). Our approach leverages the powerful capabilities of LLMs to dynamically generate an unlimited array of phrase-level class concepts. To address the hallucination challenge, we introduce an entropy-based scoring process that incorporates a ``goodness" concept selection mechanism, ensuring that only the most transferable and discriminative concepts are selected. Our InfZSL framework not only demonstrates significant improvements on three popular benchmark datasets but also generates highly interpretable, image-grounded concepts. Code will be released upon acceptance.