Asymptotic Learnability of Reinforcement Problems with Arbitrary Dependence

TL;DR

This paper investigates the conditions under which reinforcement learning agents can achieve optimal long-term rewards in environments with arbitrary stochastic dependence, extending classical assumptions like Markovianity.

Contribution

It establishes sufficient conditions for the existence of agents that attain optimal asymptotic rewards across a class of complex, dependent environments, broadening RL theory.

Findings

Identifies conditions ensuring asymptotic learnability in dependent environments

Analyzes the relationship between these conditions and existing probabilistic assumptions

Provides insights into the limits of reinforcement learning in non-Markovian settings

Abstract

We address the problem of reinforcement learning in which observations may exhibit an arbitrary form of stochastic dependence on past observations and actions. The task for an agent is to attain the best possible asymptotic reward where the true generating environment is unknown but belongs to a known countable family of environments. We find some sufficient conditions on the class of environments under which an agent exists which attains the best asymptotic reward for any environment in the class. We analyze how tight these conditions are and how they relate to different probabilistic assumptions known in reinforcement learning and related fields, such as Markov Decision Processes and mixing conditions.