Learning the Optimal Recommendation from Explorative Users

TL;DR

This paper models a more realistic, explorative user behavior in recommendation systems and studies the challenges and limits of learning optimal recommendations within a bandit framework.

Contribution

It introduces a new user interaction model and analyzes the complexity of learning optimal recommendations under this realistic setting.

Findings

Efficient learning is possible but more difficult with explorative users.

The system can identify the best recommendation with probability 1−δ within O(1/δ) interactions.

This rate is proven to be tight, showing a fundamental cost compared to fixed reward feedback.

Abstract

We propose a new problem setting to study the sequential interactions between a recommender system and a user. Instead of assuming the user is omniscient, static, and explicit, as the classical practice does, we sketch a more realistic user behavior model, under which the user: 1) rejects recommendations if they are clearly worse than others; 2) updates her utility estimation based on rewards from her accepted recommendations; 3) withholds realized rewards from the system. We formulate the interactions between the system and such an explorative user in a $K$-armed bandit framework and study the problem of learning the optimal recommendation on the system side. We show that efficient system learning is still possible but is more difficult. In particular, the system can identify the best arm with probability at least $1-\delta$ within $O(1/\delta)$ interactions, and we prove this is tight. Our finding contrasts the result for the problem of best arm identification with fixed confidence, in which the best arm can be identified with probability $1-\delta$ within $O(\log(1/\delta))$ interactions. This gap illustrates the inevitable cost the system has to pay when it learns from an explorative user's revealed preferences on its recommendations rather than from the realized rewards.