Learning Optimal Deterministic Policies with Stochastic Policy Gradients

TL;DR

This paper investigates how stochastic policy gradients can be used to learn deterministic policies in reinforcement learning, providing theoretical insights, convergence guarantees, and exploration strategies to optimize practical deployment.

Contribution

It introduces a new framework for understanding the practice of learning deterministic policies via stochastic gradients and analyzes convergence and exploration trade-offs.

Findings

Global convergence to optimal deterministic policies under certain conditions

Tuning exploration levels balances sample complexity and policy performance

Comparison of action-based and parameter-based exploration methods

Abstract

Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. They learn stochastic parametric (hyper)policies by either exploring in the space of actions or in the space of parameters. Stochastic controllers, however, are often undesirable from a practical perspective because of their lack of robustness, safety, and traceability. In common practice, stochastic (hyper)policies are learned only to deploy their deterministic version. In this paper, we make a step towards the theoretical understanding of this practice. After introducing a novel framework for modeling this scenario, we study the global convergence to the best deterministic policy, under (weak) gradient domination assumptions. Then, we illustrate how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy. Finally, we quantitatively compare action-based and parameter-based exploration, giving a formal guise to intuitive results.