General Discounting versus Average Reward

TL;DR

This paper compares average reward and discounted reward in arbitrary, non-Markovian environments, establishing conditions under which their long-term limits are equivalent or imply each other.

Contribution

It provides a theoretical analysis of the relationship between average and discounted rewards for general reward sequences and discounting schemes, beyond traditional MDP assumptions.

Findings

Asymptotic equivalence of average and discounted rewards under certain conditions.

Conditions under which the existence of one limit implies the existence of the other.

Analysis of how the effective horizon growth rate affects the relationship between the two reward measures.

Abstract

Consider an agent interacting with an environment in cycles. In every interaction cycle the agent is rewarded for its performance. We compare the average reward U from cycle 1 to m (average value) with the future discounted reward V from cycle k to infinity (discounted value). We consider essentially arbitrary (non-geometric) discount sequences and arbitrary reward sequences (non-MDP environments). We show that asymptotically U for m->infinity and V for k->infinity are equal, provided both limits exist. Further, if the effective horizon grows linearly with k or faster, then existence of the limit of U implies that the limit of V exists. Conversely, if the effective horizon grows linearly with k or slower, then existence of the limit of V implies that the limit of U exists.