TL;DR
This paper introduces a modular Transformer-based re-ranking model for long documents that models full query-to-document interaction, overcoming information bottlenecks of previous chunk-based methods.
Contribution
It proposes a novel attention-based re-ranking framework that jointly encodes query and document chunks, improving long document relevance ranking.
Findings
Effective re-ranking on Robust04 and ClueWeb09 datasets.
Superior performance on MS-MARCO document ranking.
Models utilize full query-document interaction for better relevance assessment.
Abstract
Long document re-ranking has been a challenging problem for neural re-rankers based on deep language models like BERT. Early work breaks the documents into short passage-like chunks. These chunks are independently mapped to scalar scores or latent vectors, which are then pooled into a final relevance score. These encode-and-pool methods however inevitably introduce an information bottleneck: the low dimension representations. In this paper, we propose instead to model full query-to-document interaction, leveraging the attention operation and modular Transformer re-ranker framework. First, document chunks are encoded independently with an encoder module. An interaction module then encodes the query and performs joint attention from the query to all document chunk representations. We demonstrate that the model can use this new degree of freedom to aggregate important information from the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Multi-Head Attention · Attention Is All You Need · Linear Layer · Linear Warmup With Linear Decay · Byte Pair Encoding · Absolute Position Encodings · Residual Connection · Attention Dropout · Weight Decay
