Style-Preserving Policy Optimization for Game Agents

TL;DR

This paper introduces MPPO, a reinforcement learning method that enhances game agents' proficiency while maintaining their diverse play styles, leading to more engaging and replayable gaming experiences.

Contribution

The paper proposes MPPO, a novel policy optimization technique that improves agent proficiency without sacrificing style diversity, bridging the gap between RL and evolution-based methods.

Findings

MPPO achieves proficiency comparable or superior to pure online algorithms.

MPPO preserves the diverse play styles of demonstrator agents.

Empirical results validate MPPO's effectiveness across various environments.

Abstract

Proficient game agents with diverse play styles enrich the gaming experience and enhance the replay value of games. However, recent advancements in game AI based on reinforcement learning have predominantly focused on improving proficiency, whereas methods based on evolution algorithms generate agents with diverse play styles but exhibit subpar performance compared to RL methods. To address this gap, this paper proposes Mixed Proximal Policy Optimization (MPPO), a method designed to improve the proficiency of existing suboptimal agents while retaining their distinct styles. MPPO unifies loss objectives for both online and offline samples and introduces an implicit constraint to approximate demonstrator policies by adjusting the empirical distribution of samples. Empirical results across environments of varying scales demonstrate that MPPO achieves proficiency levels comparable to, or even superior to, pure online algorithms while preserving demonstrators' play styles. This work presents an effective approach for generating highly proficient and diverse game agents, ultimately contributing to more engaging gameplay experiences.