Learning Rank Functionals: An Empirical Study

TL;DR

This paper explores the design of effective learning to rank algorithms by examining data representation, rank functionals, and loss functions, with experiments on question answering and web retrieval.

Contribution

It provides a comprehensive empirical analysis of different loss functions and modeling choices in learning to rank, including novel derivations using Markov chain theory.

Findings

Smooth rank metric approximation improves ranking quality.

Decomposition of loss functions enhances training efficiency.

Markov chain-based derivations offer new insights into loss design.

Abstract

Ranking is a key aspect of many applications, such as information retrieval, question answering, ad placement and recommender systems. Learning to rank has the goal of estimating a ranking model automatically from training data. In practical settings, the task often reduces to estimating a rank functional of an object with respect to a query. In this paper, we investigate key issues in designing an effective learning to rank algorithm. These include data representation, the choice of rank functionals, the design of the loss function so that it is correlated with the rank metrics used in evaluation. For the loss function, we study three techniques: approximating the rank metric by a smooth function, decomposition of the loss into a weighted sum of element-wise losses and into a weighted sum of pairwise losses. We then present derivations of piecewise losses using the theory of high-order Markov chains and Markov random fields. In experiments, we evaluate these design aspects on two tasks: answer ranking in a Social Question Answering site, and Web Information Retrieval.