Reinforcement Learning in Reward-Mixing MDPs

TL;DR

This paper introduces the first polynomial-time reinforcement learning algorithm for reward-mixing MDPs that operates efficiently without assumptions, even in partially observable environments with hidden reward models.

Contribution

It presents a novel, assumption-free, polynomial-time algorithm for near-optimal policy learning in reward-mixing MDPs with partial observability.

Findings

First efficient algorithm for reward-mixing MDPs without assumptions

Achieves near-optimal policy with polynomial episodes

Handles partial observability with smaller observation space

Abstract

Learning a near optimal policy in a partially observable system remains an elusive challenge in contemporary reinforcement learning. In this work, we consider episodic reinforcement learning in a reward-mixing Markov decision process (MDP). There, a reward function is drawn from one of multiple possible reward models at the beginning of every episode, but the identity of the chosen reward model is not revealed to the agent. Hence, the latent state space, for which the dynamics are Markovian, is not given to the agent. We study the problem of learning a near optimal policy for two reward-mixing MDPs. Unlike existing approaches that rely on strong assumptions on the dynamics, we make no assumptions and study the problem in full generality. Indeed, with no further assumptions, even for two switching reward-models, the problem requires several new ideas beyond existing algorithmic and analysis techniques for efficient exploration. We provide the first polynomial-time algorithm that finds an $\epsilon$-optimal policy after exploring $\tilde{O}(poly(H,\epsilon^{-1}) \cdot S^2 A^2)$ episodes, where $H$ is time-horizon and $S, A$ are the number of states and actions respectively. This is the first efficient algorithm that does not require any assumptions in partially observed environments where the observation space is smaller than the latent state space.