Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

TL;DR

This paper introduces a training-free, attention-guided calibration method to reduce position bias in multi-image retrieval, significantly improving accuracy and permutation invariance in multimodal models.

Contribution

It reveals Logit-Attention Divergence as a fundamental bias and proposes a novel, inference-time correction framework leveraging attention signals for better retrieval performance.

Findings

Achieves over 40% accuracy improvement on MS-COCO benchmarks.

Substantially enhances permutation invariance in multi-image retrieval.

Outperforms existing calibration methods with minimal computational overhead.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40\% compared to baselines. Code is available at https://github.com/brightXian/LAD.