Analysis of Agent Expertise in Ms. Pac-Man using Value-of-Information-based Policies

TL;DR

This paper introduces an information-theoretic approach to reinforcement learning in Ms. Pac-Man, controlling policy complexity to balance risk and reward, and demonstrates its effectiveness and influence on gameplay styles.

Contribution

It presents a novel value-of-information-based policy framework that manages the exploration-exploitation trade-off through policy complexity, applied to a stochastic Ms. Pac-Man environment.

Findings

Different policy complexities lead to distinct gameplay styles.

The proposed method outperforms other search mechanisms in efficiency.

Policy complexity tuning influences risk-taking and reward outcomes.

Abstract

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity. We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcement-learning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.

Full text

See pages 1-last of Sledge-TCIAIG-2017-1col-6.pdf