On Reward Structures of Markov Decision Processes

TL;DR

This paper explores the structure of Markov decision processes focusing on reward functions, introduces new estimators and theoretical insights for reinforcement learning, and proposes methods for safe and multi-objective policy optimization.

Contribution

It presents a novel estimator with instance-specific error bounds, refines key MDP constants for reward-based analysis, and develops algorithms for safe and Pareto-optimal policy planning.

Findings

New estimator with $ ilde{O}(rac{ au_s}{n})$ error bound

Theoretical link between reward shaping and learning speed

Modified algorithms for safe and multi-objective reinforcement learning

Abstract

A Markov decision process can be parameterized by a transition kernel and a reward function. Both play essential roles in the study of reinforcement learning as evidenced by their presence in the Bellman equations. In our inquiry of various kinds of "costs" associated with reinforcement learning inspired by the demands in robotic applications, rewards are central to understanding the structure of a Markov decision process and reward-centric notions can elucidate important concepts in reinforcement learning. Specifically, we study the sample complexity of policy evaluation and develop a novel estimator with an instance-specific error bound of $\tilde{O}(\sqrt{\frac{\tau_s}{n}})$ for estimating a single state value. Under the online regret minimization setting, we refine the transition-based MDP constant, diameter, into a reward-based constant, maximum expected hitting cost, and with it, provide a theoretical explanation for how a well-known technique, potential-based reward shaping, could accelerate learning with expert knowledge. In an attempt to study safe reinforcement learning, we model hazardous environments with irrecoverability and proposed a quantitative notion of safe learning via reset efficiency. In this setting, we modify a classic algorithm to account for resets achieving promising preliminary numerical results. Lastly, for MDPs with multiple reward functions, we develop a planning algorithm that computationally efficiently finds Pareto-optimal stochastic policies.