Three Methods for Training on Bandit Feedback

TL;DR

This paper reviews three distinct methods for training recommender systems on bandit feedback, comparing their principles, assumptions, and performance through experiments in a simulated environment.

Contribution

It provides a comprehensive review and empirical comparison of three different training methods for bandit feedback in recommender systems, highlighting their theoretical foundations and practical performance.

Findings

Model-based reward prediction obeys statistical principles.

Error adjustment method helps when models underfit.

Inverse propensity scoring can estimate decision rule performance.

Abstract

There are three quite distinct ways to train a machine learning model on recommender system logs. The first method is to model the reward prediction for each possible recommendation to the user, at the scoring time the best recommendation is found by computing an argmax over the personalized recommendations. This method obeys principles such as the conditionality principle and the likelihood principle. A second method is useful when the model does not fit reality and underfits. In this case, we can use the fact that we know the distribution of historical recommendations (concentrated on previously identified good actions with some exploration) to adjust the errors in the fit to be evenly distributed over all actions. Finally, the inverse propensity score can be used to produce an estimate of the decision rules expected performance. The latter two methods violate the conditionality and likelihood principle but are shown to have good performance in certain settings. In this paper we review the literature around this fundamental, yet often overlooked choice and do some experiments using the RecoGym simulation environment.