Convergence Guarantees for Deep Epsilon Greedy Policy Learning

TL;DR

This paper establishes convergence guarantees and regret bounds for the Deep Epsilon Greedy policy learning method, demonstrating its effectiveness through experiments on a nonlinear RL problem with MNIST.

Contribution

It provides the first theoretical convergence guarantees and regret bounds for Deep Epsilon Greedy, along with empirical validation on a real-world dataset.

Findings

Deep Epsilon Greedy converges under certain conditions.

Regret is minimized with cubic root exploration.

Convergence depends on noise levels in the data.

Abstract

Policy learning is a quickly growing area. As robotics and computers control day-to-day life, their error rate needs to be minimized and controlled. There are many policy learning methods and bandit methods with provable error rates that accompany them. We show an error or regret bound and convergence of the Deep Epsilon Greedy method which chooses actions with a neural network's prediction. We also show that Epsilon Greedy method regret upper bound is minimized with cubic root exploration. In experiments with the real-world dataset MNIST, we construct a nonlinear reinforcement learning problem. We witness how with either high or low noise, some methods do and some do not converge which agrees with our proof of convergence.