Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

TL;DR

This paper introduces a sample-efficient reinforcement learning method tailored for creating human-like goalkeeping agents in football video games, demonstrating improved performance and faster training times in a commercial game setting.

Contribution

A novel sample-efficient DRL approach that leverages pre-collected data and increased network plasticity for realistic game AI development.

Findings

Agent outperforms built-in AI by 10% in ball saving rate.

Training speed is 50% faster than standard DRL methods.

Qualitative evaluation shows more human-like gameplay.

Abstract

While several high profile video games have served as testbeds for Deep Reinforcement Learning (DRL), this technique has rarely been employed by the game industry for crafting authentic AI behaviors. Previous research focuses on training super-human agents with large models, which is impractical for game studios with limited resources aiming for human-like agents. This paper proposes a sample-efficient DRL method tailored for training and fine-tuning agents in industrial settings such as the video game industry. Our method improves sample efficiency of value-based DRL by leveraging pre-collected data and increasing network plasticity. We evaluate our method training a goalkeeper agent in EA SPORTS FC 25, one of the best-selling football simulations today. Our agent outperforms the game's built-in AI by 10% in ball saving rate. Ablation studies show that our method trains agents 50% faster compared to standard DRL methods. Finally, qualitative evaluation from domain experts indicates that our approach creates more human-like gameplay compared to hand-crafted agents. As a testament to the impact of the approach, the method has been adopted for use in the most recent release of the series.