Potential-Based Shaping and Q-Value Initialization are Equivalent

TL;DR

This paper demonstrates that potential-based shaping rewards and Q-value initialization are theoretically equivalent in reinforcement learning, providing insights into their properties and suggesting simpler alternatives.

Contribution

It proves the equivalence between potential-based shaping and Q-value initialization, offering a new perspective and potential simplification for reinforcement learning algorithms.

Findings

Potential-based shaping and Q-value initialization produce identical learning updates.

Under broad policies, the behaviors of both methods are indistinguishable.

The equivalence offers insights into the efficiency and theoretical properties of shaping methods.

Abstract

Shaping has proven to be a powerful but precarious means of improving reinforcement learning performance. Ng, Harada, and Russell (1999) proposed the potential-based shaping algorithm for adding shaping rewards in a way that guarantees the learner will learn optimal behavior. In this note, we prove certain similarities between this shaping algorithm and the initialization step required for several reinforcement learning algorithms. More specifically, we prove that a reinforcement learner with initial Q-values based on the shaping algorithm's potential function make the same updates throughout learning as a learner receiving potential-based shaping rewards. We further prove that under a broad category of policies, the behavior of these two learners are indistinguishable. The comparison provides intuition on the theoretical properties of the shaping algorithm as well as a suggestion for a simpler method for capturing the algorithm's benefit. In addition, the equivalence raises previously unaddressed issues concerning the efficiency of learning with potential-based shaping.