Learning to Shoot in First Person Shooter Games by Stabilizing Actions and Clustering Rewards for Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning approach for NPCs in first-person shooter games, utilizing a novel periodic update method and a weighted reward mechanism based on hit clusters to improve shooting skills.

Contribution

It proposes a new RL update strategy and reward calculation method tailored for real-time FPS environments, addressing delayed reward challenges.

Findings

Improved NPC shooting performance in FPS games.

Effective clustering of hits enhances reward signal.

Periodic updates stabilize learning process.

Abstract

While reinforcement learning (RL) has been applied to turn-based board games for many years, more complex games involving decision-making in real-time are beginning to receive more attention. A challenge in such environments is that the time that elapses between deciding to take an action and receiving a reward based on its outcome can be longer than the interval between successive decisions. We explore this in the context of a non-player character (NPC) in a modern first-person shooter game. Such games take place in 3D environments where players, both human and computer-controlled, compete by engaging in combat and completing task objectives. We investigate the use of RL to enable NPCs to gather experience from game-play and improve their shooting skill over time from a reward signal based on the damage caused to opponents. We propose a new method for RL updates and reward calculations, in which the updates are carried out periodically, after each shooting encounter has ended, and a new weighted-reward mechanism is used which increases the reward applied to actions that lead to damaging the opponent in successive hits in what we term "hit clusters".