Combinatorial Reinforcement Learning with Preference Feedback

TL;DR

This paper introduces a novel combinatorial reinforcement learning framework with preference feedback modeled by a multinomial logistic model, addressing challenges of unknown item values and tractable assortment selection, and proposes an efficient algorithm with theoretical guarantees.

Contribution

It presents the first algorithm with statistical guarantees for combinatorial RL with preference feedback under a contextual MNL model.

Findings

MNL-VQL achieves nearly minimax-optimal regret.

First regret lower bound established for linear MDPs with MNL feedback.

Provides the first statistical guarantees in combinatorial RL with preference feedback.

Abstract

In this paper, we consider combinatorial reinforcement learning with preference feedback, where a learning agent sequentially offers an action--an assortment of multiple items to--a user, whose preference feedback follows a multinomial logistic (MNL) model. This framework allows us to model real-world scenarios, particularly those involving long-term user engagement, such as in recommender systems and online advertising. However, this framework faces two main challenges: (1) the unknown value of each item, unlike traditional MNL bandits that only address single-step preference feedback, and (2) the difficulty of ensuring optimism while maintaining tractable assortment selection in the combinatorial action space with unknown values. In this paper, we assume a contextual MNL preference model, where the mean utilities are linear, and the value of each item is approximated by a general function. We propose an algorithm, MNL-VQL, that addresses these challenges, making it both computationally and statistically efficient. As a special case, for linear MDPs (with the MNL preference feedback), we establish the first regret lower bound in this framework and show that MNL-VQL achieves nearly minimax-optimal regret. To the best of our knowledge, this is the first work to provide statistical guarantees in combinatorial RL with preference feedback.