InfoCIR: Multimedia Analysis for Composed Image Retrieval

TL;DR

InfoCIR is an interactive visual analytics system that helps users understand, diagnose, and improve composed image retrieval by integrating explainability tools, prompt engineering, and spatial reasoning in a unified dashboard.

Contribution

We introduce InfoCIR, a novel system combining retrieval, explainability, and prompt engineering for composed image retrieval in a flexible, modular interface.

Findings

Enables diagnosis of retrieval failures.

Guides effective prompt enhancement.

Accelerates insight generation during model development.

Abstract

Composed Image Retrieval (CIR) allows users to search for images by combining a reference image with a text prompt that describes desired modifications. While vision-language models like CLIP have popularized this task by embedding multiple modalities into a joint space, developers still lack tools that reveal how these multimodal prompts interact with embedding spaces and why small wording changes can dramatically alter the results. We present InfoCIR, a visual analytics system that closes this gap by coupling retrieval, explainability, and prompt engineering in a single, interactive dashboard. InfoCIR integrates a state-of-the-art CIR back-end (SEARLE arXiv:2303.15247) with a six-panel interface that (i) lets users compose image + text queries, (ii) projects the top-k results into a low-dimensional space using Uniform Manifold Approximation and Projection (UMAP) for spatial reasoning, (iii) overlays similarity-based saliency maps and gradient-derived token-attribution bars for local explanation, and (iv) employs an LLM-powered prompt enhancer that generates counterfactual variants and visualizes how these changes affect the ranking of user-selected target images. A modular architecture built on Plotly-Dash allows new models, datasets, and attribution methods to be plugged in with minimal effort. We argue that InfoCIR helps diagnose retrieval failures, guides prompt enhancement, and accelerates insight generation during model development. All source code allowing for a reproducible demo is available at https://github.com/giannhskp/InfoCIR.