Leveraging the Power of Conversations: Optimal Key Term Selection in Conversational Contextual Bandits

TL;DR

This paper introduces three novel algorithms for conversational contextual bandits that improve preference learning exploration and conversation initiation, achieving near-optimal regret bounds and significant empirical performance gains.

Contribution

The paper proposes CLiSK, CLiME, and CLiSK-ME algorithms with theoretical regret bounds and practical improvements for conversational bandit systems.

Findings

Achieve a regret upper bound of $O( oot{2}dT ext{log}T)$, tighter than previous methods.

Establish a matching lower bound of $ oot{2}dT$, showing near-minimax optimality.

Improve cumulative regret by at least 14.6% on real-world datasets.

Abstract

Conversational recommender systems proactively query users with relevant "key terms" and leverage the feedback to elicit users' preferences for personalized recommendations. Conversational contextual bandits, a prevalent approach in this domain, aim to optimize preference learning by balancing exploitation and exploration. However, several limitations hinder their effectiveness in real-world scenarios. First, existing algorithms employ key term selection strategies with insufficient exploration, often failing to thoroughly probe users' preferences and resulting in suboptimal preference estimation. Second, current algorithms typically rely on deterministic rules to initiate conversations, causing unnecessary interactions when preferences are well-understood and missed opportunities when preferences are uncertain. To address these limitations, we propose three novel algorithms: CLiSK, CLiME, and CLiSK-ME. CLiSK introduces smoothed key term contexts to enhance exploration in preference learning, CLiME adaptively initiates conversations based on preference uncertainty, and CLiSK-ME integrates both techniques. We theoretically prove that all three algorithms achieve a tighter regret upper bound of $O(\sqrt{dT\log{T}})$ with respect to the time horizon $T$, improving upon existing methods. Additionally, we provide a matching lower bound $\Omega(\sqrt{dT})$ for conversational bandits, demonstrating that our algorithms are nearly minimax optimal. Extensive evaluations on both synthetic and real-world datasets show that our approaches achieve at least a 14.6% improvement in cumulative regret.