Production Ranking Systems: A Review

TL;DR

This paper reviews production-level ranking systems, highlighting their layered architecture involving data processing, representation learning, candidate selection, and online inference, to provide insights into scalable ranking at scale.

Contribution

It offers a comprehensive overview of the layered architecture of production ranking systems, detailing the tools, algorithms, and challenges involved.

Findings

Ranking systems are multi-layered systems of systems.

Different layers employ diverse algorithms and tools.

Layered approach introduces specific challenges in real-time response.

Abstract

The problem of ranking is a multi-billion dollar problem. In this paper we present an overview of several production quality ranking systems. We show that due to conflicting goals of employing the most effective machine learning models and responding to users in real time, ranking systems have evolved into a system of systems, where each subsystem can be viewed as a component layer. We view these layers as being data processing, representation learning, candidate selection and online inference. Each layer employs different algorithms and tools, with every end-to-end ranking system spanning multiple architectures. Our goal is to familiarize the general audience with a working knowledge of ranking at scale, the tools and algorithms employed and the challenges introduced by adopting a layered approach.