SumRank: Aligning Summarization Models for Long-Document Listwise Reranking

TL;DR

SumRank is a novel summarization model designed to efficiently improve long-document reranking by compressing documents into summaries aligned with ranking objectives, achieving state-of-the-art results.

Contribution

We introduce SumRank, a three-stage training pipeline for a summarization model that enhances long-document reranking efficiency and effectiveness.

Findings

Achieves state-of-the-art ranking performance on TREC DL datasets.

Reduces summarization overhead and reranking complexity.

Improves efficiency without sacrificing accuracy.

Abstract

Large Language Models (LLMs) have demonstrated superior performance in listwise passage reranking task. However, directly applying them to rank long-form documents introduces both effectiveness and efficiency issues due to the substantially increased context length. To address this challenge, we propose a pointwise summarization model SumRank, aligned with downstream listwise reranking, to compress long-form documents into concise rank-aligned summaries before the final listwise reranking stage. To obtain our summarization model SumRank, we introduce a three-stage training pipeline comprising cold-start Supervised Fine-Tuning (SFT), specialized RL data construction, and rank-driven alignment via Reinforcement Learning. This paradigm aligns the SumRank with downstream ranking objectives to preserve relevance signals. We conduct extensive experiments on five benchmark datasets from the TREC Deep Learning tracks (TREC DL 19-23). Results show that our lightweight SumRank model achieves state-of-the-art (SOTA) ranking performance while significantly improving efficiency by reducing both summarization overhead and reranking complexity.