A scale invariant ranking function for learning-to-rank: a real-world use case

TL;DR

This paper introduces a novel scale-invariant ranking function (SIR) that enhances learning-to-rank models' robustness to feature scaling inconsistencies in real-world environments, demonstrated on large-scale hotel booking data.

Contribution

The paper proposes SIR, a new scale-invariant ranking function that improves the stability of learning-to-rank models under feature scale variations in production.

Findings

SIR improves ranking performance when feature scales are inconsistent.

SIR achieves comparable accuracy to classic models when scales are consistent.

Incorporating SIR yields up to 14.7% performance gains in adverse scenarios.

Abstract

Nowadays, Online Travel Agencies provide the main service for booking holidays, business trips, accommodations, etc. As in many e-commerce services where users, items, and preferences are involved, the use of a Recommender System facilitates the navigation of the marketplaces. One of the main challenges when productizing machine learning models (and in this case, Learning-to-Rank models) is the need of, not only consistent pre-processing transformations, but also input features maintaining a similar scale both at training and prediction time. However, the features' scale does not necessarily stay the same in the real-world production environment, which could lead to unexpected ranking order. Normalization techniques such as feature standardization, batch normalization and layer normalization are commonly used to tackle the scaling issue. However, these techniques. To address this issue, in this paper we propose a novel scale-invariant ranking function (dubbed as SIR) which is accomplished by combining a deep and a wide neural network. We incorporate SIR with five state-of-the-art Learning-to-Rank models and compare the performance of the combined models with the classic algorithms on a large data set containing 56 million booked searches from the Hotels.com website. Besides, we simulate four real-world scenarios where the features' scale at the test set is inconsistent with that at the training set. The results reveal that when the features' scale is inconsistent at prediction time, Learning-To-Rank methods incorporating SIR outperform their original counterpart in all scenarios (with performance difference up to 14.7%), while when the features' scale at the training and test set are consistent our proposal achieves comparable accuracy to the classic algorithms.