A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs

TL;DR

This paper introduces a novel harmonic mean-based operator for reinforcement learning in SMDPs, enabling accurate reward rate estimation in non-stationary environments and improving upon existing ratio-based methods.

Contribution

It proposes a new harmonic mean formulation for average reward in SMDPs that handles non-stationarity, with theoretical analysis and empirical validation.

Findings

The harmonic mean operator accurately computes reward rates under non-stationarity.

The new algorithms outperform existing ratio-based methods in non-stationary settings.

Theoretical properties of the operator are rigorously proven.

Abstract

Recent research has revived and amplified interest in algorithms for undiscounted average reward reinforcement learning in infinite-horizon, non-episodic (continuing) tasks. Semi-Markov decision processes (SMDPs) are of particular interest. In SMDPs, discrete actions stochastically generate both rewards and durations, and the objective is to optimize the average reward rate. Existing algorithms approach this by optimizing the ratio of rewards to durations. However, when rewards and durations are non-stationary (in the infinite horizon), this can be incorrect. This paper presents a novel modified harmonic mean operator that correctly computes reward rates even under such conditions. This yields model-free learning algorithms that can work with SMDPs, while maintaining robustness to non-stationary reward and duration distributions over time. We prove theoretical properties of the modified harmonic mean operator, and empirically demonstrate its efficacy in comparison to existing algorithms.