An Efficient Algorithm for Deep Stochastic Contextual Bandits

TL;DR

This paper introduces a stage-wise stochastic gradient descent algorithm for deep stochastic contextual bandits, providing convergence guarantees and demonstrating effectiveness on real-world datasets.

Contribution

It formulates the deep stochastic contextual bandit problem as a non-convex optimization and proposes a novel algorithm with proven convergence properties.

Findings

Algorithm converges to a local optimal policy with high probability.

Demonstrates superior performance on multiple real-world datasets.

Provides theoretical analysis of convergence in deep stochastic bandits.

Abstract

In stochastic contextual bandit (SCB) problems, an agent selects an action based on certain observed context to maximize the cumulative reward over iterations. Recently there have been a few studies using a deep neural network (DNN) to predict the expected reward for an action, and the DNN is trained by a stochastic gradient based method. However, convergence analysis has been greatly ignored to examine whether and where these methods converge. In this work, we formulate the SCB that uses a DNN reward function as a non-convex stochastic optimization problem, and design a stage-wise stochastic gradient descent algorithm to optimize the problem and determine the action policy. We prove that with high probability, the action sequence chosen by this algorithm converges to a greedy action policy respecting a local optimal reward function. Extensive experiments have been performed to demonstrate the effectiveness and efficiency of the proposed algorithm on multiple real-world datasets.