Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies

TL;DR

This paper introduces a rigorous benchmarking framework for reinforcement learning that constructs controlled environments with known optimal policies, enabling precise evaluation of algorithms across diverse scenarios.

Contribution

It extends converse optimality to nonlinear stochastic systems, allowing systematic generation of benchmark environments with guaranteed optimal policies.

Findings

Framework enables controlled environment generation

Demonstrates capacity for comprehensive RL evaluation

Provides a reproducible basis for benchmarking

Abstract

The objective comparison of Reinforcement Learning (RL) algorithms is notoriously complex as outcomes and benchmarking of performances of different RL approaches are critically sensitive to environmental design, reward structures, and stochasticity inherent in both algorithmic learning and environmental dynamics. To manage this complexity, we introduce a rigorous benchmarking framework by extending converse optimality to discrete-time, control-affine, nonlinear systems with noise. Our framework provides necessary and sufficient conditions, under which a prescribed value function and policy are optimal for constructed systems, enabling the systematic generation of benchmark families via homotopy variations and randomized parameters. We validate it by automatically constructing diverse environments, demonstrating our framework's capacity for a controlled and comprehensive evaluation across algorithms. By assessing standard methods against a ground-truth optimum, our work delivers a reproducible foundation for precise and rigorous RL benchmarking.