DRE-Bot: A Hierarchical First Person Shooter Bot Using Multiple Sarsa({\lambda}) Reinforcement Learners

TL;DR

This paper presents DRE-Bot, a hierarchical FPS game NPC controlled by multiple reinforcement learners using Sarsa({}) to improve decision-making through trial and error, tested against fixed strategy bots.

Contribution

Introduces a novel hierarchical architecture for FPS NPCs using multiple Sarsa({}) learners, demonstrating adaptive learning in a complex game environment.

Findings

DRE-Bot outperforms fixed strategy bots in Unreal Tournament 2004.

Varying  and  parameters affects the learning performance.

Hierarchical reinforcement learning enhances NPC adaptability.

Abstract

This paper describes an architecture for controlling non-player characters (NPC) in the First Person Shooter (FPS) game Unreal Tournament 2004. Specifically, the DRE-Bot architecture is made up of three reinforcement learners, Danger, Replenish and Explore, which use the tabular Sarsa({\lambda}) algorithm. This algorithm enables the NPC to learn through trial and error building up experience over time in an approach inspired by human learning. Experimentation is carried to measure the performance of DRE-Bot when competing against fixed strategy bots that ship with the game. The discount parameter, {\gamma}, and the trace parameter, {\lambda}, are also varied to see if their values have an effect on the performance.