Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes

TL;DR

This paper introduces QR-MAX, a novel model-based reinforcement learning algorithm for discrete non-Markovian reward decision processes that guarantees PAC convergence and improves sample efficiency by exploiting reward structure.

Contribution

The paper presents the first model-based RL algorithm for discrete NMRDPs that uses reward machine factorization for PAC guarantees and extends it to continuous spaces with a new discretizer.

Findings

QR-MAX achieves PAC convergence with polynomial sample complexity.

Bucket-QR-MAX enables fast, stable learning in continuous state spaces.

Experimental results show improved sample efficiency and robustness over state-of-the-art methods.

Abstract

Many practical decision-making problems involve tasks whose success depends on the entire system history, rather than on achieving a state with desired properties. Markovian Reinforcement Learning (RL) approaches are not suitable for such tasks, while RL with non-Markovian reward decision processes (NMRDPs) enables agents to tackle temporal-dependency tasks. This approach has long been known to lack formal guarantees on both (near-)optimality and sample efficiency. We contribute to solving both issues with QR-MAX, a novel model-based algorithm for discrete NMRDPs that factorizes Markovian transition learning from non-Markovian reward handling via reward machines. To the best of our knowledge, this is the first model-based RL algorithm for discrete-action NMRDPs that exploits this factorization to obtain PAC convergence to $\varepsilon$-optimal policies with polynomial sample complexity. We then extend QR-MAX to continuous state spaces with Bucket-QR-MAX, a SimHash-based discretiser that preserves the same factorized structure and achieves fast and stable learning without manual gridding or function approximation. We experimentally compare our method with modern state-of-the-art model-based RL approaches on environments of increasing complexity, showing a significant improvement in sample efficiency and increased robustness in finding optimal policies.