Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering

TL;DR

This paper proposes a geometry-aware CLIP retrieval method that enhances local neighborhood alignment and control, improving attribute-binding and compositional retrieval without retraining.

Contribution

It introduces neighborhood re-ranking with Hungarian matching and query-conditioned local steering to improve local structure in CLIP retrieval.

Findings

Improved retrieval performance on attribute-binding tasks.

Enhanced control over retrieval results via local neighborhood steering.

Demonstrated effectiveness without retraining the model.

Abstract

CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining.