Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders

TL;DR

The paper introduces the Set-Encoder, a permutation-invariant cross-encoder architecture for listwise passage re-ranking that effectively models inter-passage interactions while improving efficiency and robustness to passage order permutations.

Contribution

The Set-Encoder enables efficient, permutation-invariant passage interactions in listwise re-ranking models, combining effectiveness with improved computational efficiency.

Findings

Achieves state-of-the-art effectiveness on TREC Deep Learning and TIREx datasets.

More efficient than existing listwise models while maintaining effectiveness.

Robust to input passage order permutations.

Abstract

Existing cross-encoder models can be categorized as pointwise, pairwise, or listwise. Pairwise and listwise models allow passage interactions, which typically makes them more effective than pointwise models but less efficient and less robust to input passage order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while being more efficient and invariant to input passage order permutations. Compared to pointwise models, the Set-Encoder is particularly more effective when considering inter-passage information, such as novelty, and retains its advantageous properties compared to other listwise models. Our code is publicly available at https://github.com/webis-de/ECIR-25.