Performance Dynamics and Termination Errors in Reinforcement Learning: A Unifying Perspective

TL;DR

This paper analyzes the probability of premature termination errors in reinforcement learning due to stochastic reward sequences, providing mathematical insights and practical mechanisms to reduce such errors, supported by simulations.

Contribution

It offers a unifying combinatorial analysis of termination errors in reinforcement learning and proposes practical methods to mitigate these errors.

Findings

Error probability can be mathematically characterized.

Premature termination errors can be significantly high.

Practical mechanisms can effectively reduce termination errors.

Abstract

In reinforcement learning, a decision needs to be made at some point as to whether it is worthwhile to carry on with the learning process or to terminate it. In many such situations, stochastic elements are often present which govern the occurrence of rewards, with the sequential occurrences of positive rewards randomly interleaved with negative rewards. For most practical learners, the learning is considered useful if the number of positive rewards always exceeds the negative ones. A situation that often calls for learning termination is when the number of negative rewards exceeds the number of positive rewards. However, while this seems reasonable, the error of premature termination, whereby termination is enacted along with the conclusion of learning failure despite the positive rewards eventually far outnumber the negative ones, can be significant. In this paper, using combinatorial analysis we study the error probability in wrongly terminating a reinforcement learning activity which undermines the effectiveness of an optimal policy, and we show that the resultant error can be quite high. Whilst we demonstrate mathematically that such errors can never be eliminated, we propose some practical mechanisms that can effectively reduce such errors. Simulation experiments have been carried out, the results of which are in close agreement with our theoretical findings.