Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors
Leonardo Villalobos-Arias, Grant Forbes, Jianxun Wang, David L Roberts, Arnav Jhala

TL;DR
This paper investigates how intrinsic motivation influences reinforcement learning agents' behaviors, revealing that it increases initial rewards and alters gameplay, while generalized reward matching can reduce reward hacking.
Contribution
It provides an empirical evaluation of intrinsic motivation effects on RL agents and introduces generalized reward matching as a method to mitigate reward hacking.
Findings
Intrinsic motivation increases initial rewards.
IM alters agent gameplay behavior.
GRM reduces reward hacking in some scenarios.
Abstract
Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
