Generate, Transduct, Adapt: Iterative Transduction with VLMs

TL;DR

This paper introduces GTA-CLIP, an iterative transductive method that leverages language models to improve zero-shot and few-shot classification accuracy with vision-language models like CLIP.

Contribution

GTA-CLIP is a novel iterative approach that incorporates language supervision into transductive zero-shot learning, enhancing performance over existing methods.

Findings

Achieves an average of 8.6% and 3.7% improvement across datasets.

Demonstrates effectiveness in both zero-shot and few-shot settings.

Ablation studies confirm the importance of each iterative step.

Abstract

Transductive zero-shot learning with vision-language models leverages image-image similarities within the dataset to achieve better classification accuracy compared to the inductive setting. However, there is little work that explores the structure of the language space in this context. We propose GTA-CLIP, a novel technique that incorporates supervision from language models for joint transduction in language and vision spaces. Our approach is iterative and consists of three steps: (i) incrementally exploring the attribute space by querying language models, (ii) an attribute-augmented transductive inference procedure, and (iii) fine-tuning the language and vision encoders based on inferred labels within the dataset. Through experiments with CLIP encoders, we demonstrate that GTA-CLIP, yields an average performance improvement of 8.6% and 3.7% across 12 datasets and 3 encoders, over CLIP and transductive CLIP respectively in the zero-shot setting. We also observe similar improvements in a few-shot setting. We present ablation studies that demonstrate the value of each step and visualize how the vision and language spaces evolve over iterations driven by the transductive learning. Code is released at https://github.com/cvl-umass/GTA-CLIP