Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

TL;DR

This paper introduces the Poly-encoder, a new transformer architecture that balances speed and accuracy for multi-sentence scoring tasks, outperforming existing models through optimized pre-training and fine-tuning strategies.

Contribution

The paper proposes the Poly-encoder architecture, which learns global self-attention features, and demonstrates its effectiveness and efficiency over traditional Cross- and Bi-encoders.

Findings

Poly-encoders achieve state-of-the-art results on three tasks.

Poly-encoders are faster than Cross-encoders.

Poly-encoders are more accurate than Bi-encoders.

Abstract

The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.