Policy Certificates: Towards Accountable Reinforcement Learning

TL;DR

This paper introduces policy certificates for reinforcement learning, providing guarantees on policy quality to enhance accountability, especially in high-stakes settings, and presents algorithms with theoretical guarantees and improved sample efficiency.

Contribution

It proposes a novel framework for policy certificates in RL, introduces two new algorithms with certificates, and offers theoretical analysis ensuring policy quality and sample efficiency.

Findings

Certificates can improve sample efficiency in tabular MDPs

First algorithms to achieve minimax-optimal PAC bounds with certificates

Matching or surpassing existing minimax regret bounds

Abstract

The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration. Existing algorithms provide little information about the quality of their current policy before executing it, and thus have limited use in high-stakes applications like healthcare. We address this lack of accountability by proposing that algorithms output policy certificates. These certificates bound the sub-optimality and return of the policy in the next episode, allowing humans to intervene when the certified quality is not satisfactory. We further introduce two new algorithms with certificates and present a new framework for theoretical analysis that guarantees the quality of their policies and certificates. For tabular MDPs, we show that computing certificates can even improve the sample-efficiency of optimism-based exploration. As a result, one of our algorithms is the first to achieve minimax-optimal PAC bounds up to lower-order terms, and this algorithm also matches (and in some settings slightly improves upon) existing minimax regret bounds.