Neural Risk-sensitive Satisficing in Contextual Bandits

TL;DR

This paper introduces NeuralRS, a neural network-based algorithm for contextual bandits that improves upon previous linear methods by handling complex, non-linear reward relationships in recommendation systems.

Contribution

It extends the RegLinRS algorithm by integrating neural networks, enabling better performance in environments with non-linear feature-reward relationships.

Findings

NeuralRS effectively models non-linear reward functions.

NeuralRS outperforms linear methods in complex environments.

The approach demonstrates improved adaptability in recommendation tasks.

Abstract

The contextual bandit problem, which is a type of reinforcement learning tasks, provides an effective framework for solving challenges in recommendation systems, such as satisfying real-time requirements, enabling personalization, addressing cold-start problems. However, contextual bandit algorithms face challenges since they need to handle large state-action spaces sequentially. These challenges include the high costs for learning and balancing exploration and exploitation, as well as large variations in performance that depend on the domain of application. To address these challenges, Tsuboya et~al. proposed the Regional Linear Risk-sensitive Satisficing (RegLinRS) algorithm. RegLinRS switches between exploration and exploitation based on how well the agent has achieved the target. However, the reward expectations in RegLinRS are linearly approximated based on features, which limits its applicability when the relationship between features and reward expectations is non-linear. To handle more complex environments, we proposed Neural Risk-sensitive Satisficing (NeuralRS), which incorporates neural networks into RegLinRS, and demonstrated its utility.