ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval

TL;DR

This paper introduces ADaFuSE, a novel adaptive fusion model for diffusion-based text-to-image retrieval that improves performance and robustness by dynamically balancing and calibrating multi-modal information without altering existing frameworks.

Contribution

The paper proposes ADaFuSE, a lightweight, adaptive fusion mechanism with semantic-aware experts that enhances diffusion-augmented I-TIR by effectively aligning multi-modal views and reducing noise impact.

Findings

Achieves state-of-the-art performance on four I-TIR benchmarks.

Surpasses DAR by up to 3.49% in Hits@10 with minimal parameter increase.

Demonstrates robustness to noisy and longer interactive queries.

Abstract

Recent advances in interactive text-to-image retrieval (I-TIR) use diffusion models to bridge the modality gap between the textual information need and the images to be searched, resulting in increased effectiveness. However, existing frameworks fuse multi-modal views of user feedback by simple embedding addition. In this work, we show that this static and undifferentiated fusion indiscriminately incorporates generative noise produced by the diffusion model, leading to performance degradation for up to 55.62% samples. We further propose ADaFuSE (Adaptive Diffusion-Text Fusion with Semantic-aware Experts), a lightweight fusion model designed to align and calibrate multi-modal views for diffusion-augmented I-TIR, which can be plugged into existing frameworks without modifying the backbone encoder. Specifically, we introduce a dual-branch fusion mechanism that employs an adaptive gating branch to dynamically balance modality reliability, alongside a semantic-aware mixture-of-experts branch to capture fine-grained cross-modal nuances. Via thorough evaluation over four standard I-TIR benchmarks, ADaFuSE achieves state-of-the-art performance, surpassing DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase, while exhibiting stronger robustness to noisy and longer interactive queries. These results show that generative augmentation coupled with principled fusion provides a simple, generalizable alternative to fine-tuning for interactive retrieval.