Fast Slate Policy Optimization: Going Beyond Plackett-Luce

TL;DR

This paper introduces a new policy optimization method for large-scale slate decision systems, enabling efficient learning in massive action spaces beyond traditional models like Plackett-Luce.

Contribution

It proposes a novel relaxation of decision functions leading to a scalable and effective policy optimization algorithm for large action spaces.

Findings

Outperforms Plackett-Luce in large action space scenarios

Scales efficiently to millions of actions

Demonstrates improved reward optimization

Abstract

An increasingly important building block of large scale machine learning systems is based on returning slates; an ordered lists of items given a query. Applications of this technology include: search, information retrieval and recommender systems. When the action space is large, decision systems are restricted to a particular structure to complete online queries quickly. This paper addresses the optimization of these large scale decision systems given an arbitrary reward function. We cast this learning problem in a policy optimization framework and propose a new class of policies, born from a novel relaxation of decision functions. This results in a simple, yet efficient learning algorithm that scales to massive action spaces. We compare our method to the commonly adopted Plackett-Luce policy class and demonstrate the effectiveness of our approach on problems with action space sizes in the order of millions.