Learning to Truncate Ranked Lists for Information Retrieval

TL;DR

This paper introduces AttnCut, a transformer-based model that directly optimizes user-defined objectives for ranked list truncation in information retrieval, outperforming existing sequential decision methods.

Contribution

The paper presents a novel global decision model for list truncation using transformers and RAML, directly optimizing practical user-defined objectives.

Findings

AttnCut outperforms state-of-the-art baselines on Robust04 and MQ2007 datasets.

Direct optimization of user objectives improves truncation effectiveness.

Transformer architecture captures global dependencies in ranked lists.

Abstract

Ranked list truncation is of critical importance in a variety of professional information retrieval applications such as patent search or legal search. The goal is to dynamically determine the number of returned documents according to some user-defined objectives, in order to reach a balance between the overall utility of the results and user efforts. Existing methods formulate this task as a sequential decision problem and take some pre-defined loss as a proxy objective, which suffers from the limitation of local decision and non-direct optimization. In this work, we propose a global decision based truncation model named AttnCut, which directly optimizes user-defined objectives for the ranked list truncation. Specifically, we take the successful transformer architecture to capture the global dependency within the ranked list for truncation decision, and employ the reward augmented maximum likelihood (RAML) for direct optimization. We consider two types of user-defined objectives which are of practical usage. One is the widely adopted metric such as F1 which acts as a balanced objective, and the other is the best F1 under some minimal recall constraint which represents a typical objective in professional search. Empirical results over the Robust04 and MQ2007 datasets demonstrate the effectiveness of our approach as compared with the state-of-the-art baselines.