Human-level performance in first-person multiplayer games with population-based deep reinforcement learning

TL;DR

This paper presents a population-based deep reinforcement learning approach enabling agents to achieve human-level performance in a complex multiplayer 3D first-person game, demonstrating advanced behaviors and surpassing human players.

Contribution

Introduces a novel two-tier optimization process with population-based RL agents that learn multi-timescale reasoning and achieve human-level performance in multiplayer 3D environments.

Findings

Agents exceeded strong human win-rates as teammates and opponents

Achieved human-like behaviors such as navigation and defending

Outperformed existing state-of-the-art agents

Abstract

Recent progress in artificial intelligence through reinforcement learning (RL) has shown great success on increasingly complex single-agent environments and two-player turn-based games. However, the real-world contains multiple agents, each learning and acting independently to cooperate and compete with other agents, and environments reflecting this degree of complexity remain an open challenge. In this work, we demonstrate for the first time that an agent can achieve human-level in a popular 3D multiplayer first-person video game, Quake III Arena Capture the Flag, using only pixels and game points as input. These results were achieved by a novel two-tier optimisation process in which a population of independent RL agents are trained concurrently from thousands of parallel matches with agents playing in teams together and against each other on randomly generated environments. Each agent in the population learns its own internal reward signal to complement the sparse delayed reward from winning, and selects actions using a novel temporally hierarchical representation that enables the agent to reason at multiple timescales. During game-play, these agents display human-like behaviours such as navigating, following, and defending based on a rich learned representation that is shown to encode high-level game knowledge. In an extensive tournament-style evaluation the trained agents exceeded the win-rate of strong human players both as teammates and opponents, and proved far stronger than existing state-of-the-art agents. These results demonstrate a significant jump in the capabilities of artificial agents, bringing us closer to the goal of human-level intelligence.