MICE: Minimal Interaction Cross-Encoders for efficient Re-ranking

TL;DR

This paper introduces MICE, a new architecture that reduces cross-encoder inference latency by removing unnecessary interactions, achieving efficiency comparable to late-interaction models while maintaining high effectiveness and better generalization.

Contribution

MICE is a novel architecture that bridges cross-encoders and late-interaction models by minimizing interactions, significantly reducing inference latency while preserving effectiveness.

Findings

MICE reduces inference latency fourfold compared to standard cross-encoders.

MICE matches late-interaction models like ColBERT in efficiency.

MICE retains most of cross-encoder effectiveness and generalizes well out-of-domain.

Abstract

Cross-encoders deliver state-of-the-art ranking effectiveness in information retrieval, but have a high inference cost. This prevents them from being used as first-stage rankers, but also incurs a cost when re-ranking documents. Prior work has addressed this bottleneck from two largely separate directions: accelerating cross-encoder inference by sparsifying the attention process or improving first-stage retrieval effectiveness using more complex models, e.g. late-interaction ones. In this work, we propose to bridge these two approaches, based on an in-depth understanding of the internal mechanisms of cross-encoders. Starting from cross-encoders, we show that it is possible to derive a new late-interaction-like architecture by carefully removing detrimental or unnecessary interactions. We name this architecture MICE (Minimal Interaction Cross-Encoders). We extensively evaluate MICE across both in-domain (ID) and out-of-domain (OOD) datasets. MICE decreases fourfold the inference latency compared to standard cross-encoders, matching late-interaction models like ColBERT while retaining most of cross-encoder ID effectiveness and demonstrating superior generalization abilities in OOD.