Local Policy Improvement for Recommender Systems

TL;DR

This paper introduces a local policy improvement method for recommender systems that optimizes a lower bound of expected reward without off-policy correction, suitable for frequent policy updates and data reuse.

Contribution

It proposes a novel local policy improvement approach that avoids importance sampling, enabling more practical and efficient policy updates in recommender systems.

Findings

Method effectively improves policies in sequential recommendation tasks.

Avoids the practical limitations of importance sampling correction.

Provides empirical evidence and practical guidelines for implementation.

Abstract

Recommender systems predict what items a user will interact with next, based on their past interactions. The problem is often approached through supervised learning, but recent advancements have shifted towards policy optimization of rewards (e.g., user engagement). One challenge with the latter is policy mismatch: we are only able to train a new policy given data collected from a previously-deployed policy. The conventional way to address this problem is through importance sampling correction, but this comes with practical limitations. We suggest an alternative approach of local policy improvement without off-policy correction. Our method computes and optimizes a lower bound of expected reward of the target policy, which is easy to estimate from data and does not involve density ratios (such as those appearing in importance sampling correction). This local policy improvement paradigm is ideal for recommender systems, as previous policies are typically of decent quality and policies are updated frequently. We provide empirical evidence and practical recipes for applying our technique in a sequential recommendation setting.