Rationality Measurement and Theory for Reinforcement Learning Agents

TL;DR

This paper introduces rationality measures and a theoretical framework for reinforcement learning agents, analyzing their performance and generalisability in dynamic environments.

Contribution

It defines new metrics for rationality in RL, decomposes rational risk gap, and links it to environment shifts and model complexity, supported by experiments.

Findings

Rational risk gap decomposes into environment shift and algorithmic generalisability.

Regularisers and domain randomisation can improve rationality.

Environment shifts can harm RL agent performance.

Abstract

This paper proposes a suite of rationality measures and associated theory for reinforcement learning agents, a property increasingly critical yet rarely explored. We define an action in deployment to be perfectly rational if it maximises the hidden true value function in the steepest direction. The expected value discrepancy of a policy's actions against their rational counterparts, culminating over the trajectory in deployment, is defined to be expected rational risk; an empirical average version in training is also defined. Their difference, termed as rational risk gap, is decomposed into (1) an extrinsic component caused by environment shifts between training and deployment, and (2) an intrinsic one due to the algorithm's generalisability in a dynamic environment. They are upper bounded by, respectively, (1) the $1$-Wasserstein distance between transition kernels and initial state distributions in training and deployment, and (2) the empirical Rademacher complexity of the value function class. Our theory suggests hypotheses on the benefits from regularisers (including layer normalisation, $\ell_2$ regularisation, and weight normalisation) and domain randomisation, as well as the harm from environment shifts. Experiments are in full agreement with these hypotheses. The code is available at https://github.com/EVIEHub/Rationality.