Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks

TL;DR

This paper investigates how different reinforcement learning algorithms affect policy entropy in personalization tasks, revealing that policy optimization tends to produce low-entropy policies while Q-Learning maintains higher entropy, impacting real-world applications.

Contribution

It provides a comparative analysis of policy entropy behaviors in policy optimization and Q-Learning agents, supported by experiments and theoretical insights.

Findings

Policy optimization agents often have low-entropy policies during training.

Q-Learning agents tend to maintain high-entropy policies.

Differences in entropy are due to the underlying learning algorithms.

Abstract

This effort is focused on examining the behavior of reinforcement learning systems in personalization environments and detailing the differences in policy entropy associated with the type of learning algorithm utilized. We demonstrate that Policy Optimization agents often possess low-entropy policies during training, which in practice results in agents prioritizing certain actions and avoiding others. Conversely, we also show that Q-Learning agents are far less susceptible to such behavior and generally maintain high-entropy policies throughout training, which is often preferable in real-world applications. We provide a wide range of numerical experiments as well as theoretical justification to show that these differences in entropy are due to the type of learning being employed.