Longformer for MS MARCO Document Re-ranking Task

TL;DR

This paper explores the use of Longformer, a transformer model designed for long documents, to improve neural re-ranking performance on the MS MARCO document retrieval task, demonstrating its effectiveness over traditional models.

Contribution

The paper introduces Longformer for MS MARCO document re-ranking, adapting a long-document transformer to enhance neural re-ranking accuracy.

Findings

Longformer outperforms BERT in re-ranking accuracy

Effective handling of long documents improves retrieval results

Code implementation is publicly available for reproducibility

Abstract

Two step document ranking, where the initial retrieval is done by a classical information retrieval method, followed by neural re-ranking model, is the new standard. The best performance is achieved by using transformer-based models as re-rankers, e.g., BERT. We employ Longformer, a BERT-like model for long documents, on the MS MARCO document re-ranking task. The complete code used for training the model can be found on: https://github.com/isekulic/longformer-marco