Optimize What You Evaluate With: A Simple Yet Effective Framework For Direct Optimization Of IR Metrics

TL;DR

This paper introduces a simple, effective framework for directly optimizing IR metrics like AP and nDCG by using a novel differentiable rank position function, enabling better learning-to-rank performance.

Contribution

The paper proposes a twin-sigmoid function for exact differentiable rank positions, allowing direct optimization of IR metrics with neural networks, addressing longstanding challenges.

Findings

Improved ranking performance on MSLRWEB30K benchmark

Effective gradient modification strategies for IR metric optimization

Demonstrated superiority over traditional approximation methods

Abstract

Learning-to-rank has been intensively studied and has shown significantly increasing values in a wide range of domains. The performance of learning-to-rank methods is commonly evaluated using rank-sensitive metrics, such as average precision (AP) and normalized discounted cumulative gain (nDCG). Unfortunately, how to effectively optimize rank-sensitive objectives is far from being resolved, which has been an open problem since the dawn of learning-to-rank over a decade ago. In this paper, we introduce a simple yet effective framework for directly optimizing information retrieval (IR) metrics. Specifically, we propose a novel twin-sigmoid function for deriving the exact rank positions of documents during the optimization process instead of using approximated rank positions or relying on the traditional sorting algorithms. Thanks to this, the rank positions are differentiable, enabling us to reformulate the widely used IR metrics as differentiable ones and directly optimize them based on neural networks. Furthermore, by carrying out an in-depth analysis of the gradients, we pinpoint the potential limitations inherent with direct optimization of IR metrics based on the vanilla sigmoid. To break the limitations, we propose different strategies by explicitly modifying the gradient computation. To validate the effectiveness of the proposed framework for direct optimization of IR metrics, we conduct a series of experiments on the widely used benchmark collection MSLRWEB30K.