ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectives

TL;DR

ShinRL is an open-source library that enables comprehensive evaluation of reinforcement learning algorithms from both theoretical and practical perspectives, facilitating analysis of algorithm behaviors and validation of recent theoretical insights.

Contribution

ShinRL introduces a unified environment and solver interface for analyzing RL algorithms' behaviors and validating theoretical findings, bridging practical and theoretical evaluation.

Findings

Enables analysis of the gap between learned and optimal Q values.

Facilitates empirical validation of theoretical results like KL regularization effects.

Supports consistent evaluation of both classical and deep RL algorithms.

Abstract

We present ShinRL, an open-source library specialized for the evaluation of reinforcement learning (RL) algorithms from both theoretical and practical perspectives. Existing RL libraries typically allow users to evaluate practical performances of deep RL algorithms through returns. Nevertheless, these libraries are not necessarily useful for analyzing if the algorithms perform as theoretically expected, such as if Q learning really achieves the optimal Q function. In contrast, ShinRL provides an RL environment interface that can compute metrics for delving into the behaviors of RL algorithms, such as the gap between learned and the optimal Q values and state visitation frequencies. In addition, we introduce a flexible solver interface for evaluating both theoretically justified algorithms (e.g., dynamic programming and tabular RL) and practically effective ones (i.e., deep RL, typically with some additional extensions and regularizations) in a consistent fashion. As a case study, we show that how combining these two features of ShinRL makes it easier to analyze the behavior of deep Q learning. Furthermore, we demonstrate that ShinRL can be used to empirically validate recent theoretical findings such as the effect of KL regularization for value iteration and for deep Q learning, and the robustness of entropy-regularized policies to adversarial rewards. The source code for ShinRL is available on GitHub: https://github.com/omron-sinicx/ShinRL.