Relating Reinforcement Learning to Dynamic Programming-Based Planning

TL;DR

This paper explores the connections between reinforcement learning and dynamic programming-based planning, analyzing their differences, similarities, and conditions for equivalence, and proposes optimizing truecost for better performance.

Contribution

It introduces a derandomized RL version, analyzes conditions for equivalence with planning methods, and advocates for optimizing truecost over arbitrary parameters.

Findings

Derandomized RL matches value iteration performance.

Conditions identified where cost minimization equals reward maximization.

Discounting can prevent goal achievement under certain conditions.

Abstract

This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.