Longformer for MS MARCO Document Re-ranking Task
Ivan Sekuli\'c, Amir Soleimani, Mohammad Aliannejadi, Fabio Crestani

TL;DR
This paper explores the use of Longformer, a transformer model designed for long documents, to improve neural re-ranking performance on the MS MARCO document retrieval task, demonstrating its effectiveness over traditional models.
Contribution
The paper introduces Longformer for MS MARCO document re-ranking, adapting a long-document transformer to enhance neural re-ranking accuracy.
Findings
Longformer outperforms BERT in re-ranking accuracy
Effective handling of long documents improves retrieval results
Code implementation is publicly available for reproducibility
Abstract
Two step document ranking, where the initial retrieval is done by a classical information retrieval method, followed by neural re-ranking model, is the new standard. The best performance is achieved by using transformer-based models as re-rankers, e.g., BERT. We employ Longformer, a BERT-like model for long documents, on the MS MARCO document re-ranking task. The complete code used for training the model can be found on: https://github.com/isekulic/longformer-marco
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Advanced Text Analysis Techniques · Natural Language Processing Techniques
MethodsLinear Layer · Adam · Softmax · How do I get a human at Expedia immediately? (2025-2026) · Refunds@Expedia|||How do I get a full refund from Expedia? · How do I complain to Expedia?*ComplainByAgent · Dense Connections · Weight Decay · Dropout · Linear Warmup With Linear Decay
