EFSA: Episodic Few-Shot Adaptation for Text-to-Image Retrieval

TL;DR

EFSA introduces a test-time adaptation framework that dynamically fine-tunes pre-trained vision-language models on retrieved candidates and synthetic captions, significantly improving open-domain text-to-image retrieval across diverse visual domains.

Contribution

This work presents EFSA, a novel episodic few-shot adaptation method that enhances pre-trained models' robustness in open-domain retrieval by domain-specific fine-tuning at test time.

Findings

Improves retrieval performance across 8 diverse visual domains.

Maintains generalization while adapting to specific query domains.

Effective in open-domain retrieval with over one million images.

Abstract

Text-to-image retrieval is a critical task for managing diverse visual content, but common benchmarks for the task rely on small, single-domain datasets that fail to capture real-world complexity. Pre-trained vision-language models tend to perform well with easy negatives but struggle with hard negatives--visually similar yet incorrect images--especially in open-domain scenarios. To address this, we introduce Episodic Few-Shot Adaptation (EFSA), a novel test-time framework that adapts pre-trained models dynamically to a query's domain by fine-tuning on top-k retrieved candidates and synthetic captions generated for them. EFSA improves performance across diverse domains while preserving generalization, as shown in evaluations on queries from eight highly distinct visual domains and an open-domain retrieval pool of over one million images. Our work highlights the potential of episodic few-shot adaptation to enhance robustness in the critical and understudied task of open-domain text-to-image retrieval.