Policy Fusion for Adaptive and Customizable Reinforcement Learning Agents

TL;DR

This paper introduces four policy fusion methods for combining pre-trained reinforcement learning policies to create adaptable, behavior-specific game agents that interact meaningfully with players, enhancing game design flexibility.

Contribution

It proposes novel policy fusion techniques and demonstrates their effectiveness in creating behavior-specific agents without extensive reward engineering.

Findings

Entropy-weighted policy fusion outperforms other methods.

Fusion methods enable behavior customization via inverse reinforcement learning.

Practical applications demonstrated in video game development.

Abstract

In this article we study the problem of training intelligent agents using Reinforcement Learning for the purpose of game development. Unlike systems built to replace human players and to achieve super-human performance, our agents aim to produce meaningful interactions with the player, and at the same time demonstrate behavioral traits as desired by game designers. We show how to combine distinct behavioral policies to obtain a meaningful "fusion" policy which comprises all these behaviors. To this end, we propose four different policy fusion methods for combining pre-trained policies. We further demonstrate how these methods can be used in combination with Inverse Reinforcement Learning in order to create intelligent agents with specific behavioral styles as chosen by game designers, without having to define many and possibly poorly-designed reward functions. Experiments on two different environments indicate that entropy-weighted policy fusion significantly outperforms all others. We provide several practical examples and use-cases for how these methods are indeed useful for video game production and designers.