Multi-Objective Intrinsic Reward Learning for Conversational Recommender Systems

TL;DR

This paper introduces a multi-objective intrinsic reward learning framework for conversational recommender systems, optimizing success rate and conversation efficiency through a bi-level optimization approach.

Contribution

It proposes a novel multi-objective bi-level optimization method to learn intrinsic rewards that improve CRS performance, addressing the limitations of handcrafted reward functions.

Findings

Significant performance improvements on three CRS benchmarks.

Effective optimization of success rate and conversation length.

Demonstrated the benefits of learned intrinsic rewards over traditional methods.

Abstract

Conversational Recommender Systems (CRS) actively elicit user preferences to generate adaptive recommendations. Mainstream reinforcement learning-based CRS solutions heavily rely on handcrafted reward functions, which may not be aligned with user intent in CRS tasks. Therefore, the design of task-specific rewards is critical to facilitate CRS policy learning, which remains largely under-explored in the literature. In this work, we propose a novel approach to address this challenge by learning intrinsic rewards from interactions with users. Specifically, we formulate intrinsic reward learning as a multi-objective bi-level optimization problem. The inner level optimizes the CRS policy augmented by the learned intrinsic rewards, while the outer level drives the intrinsic rewards to optimize two CRS-specific objectives: maximizing the success rate and minimizing the number of turns to reach a successful recommendation in conversations. To evaluate the effectiveness of our approach, we conduct extensive experiments on three public CRS benchmarks. The results show that our algorithm significantly improves CRS performance by exploiting informative learned intrinsic rewards.