Deep Q-Network for AI Soccer

TL;DR

This paper applies Deep Q-Network reinforcement learning to AI Soccer, successfully training agents that achieved a top 16 finish in an international competition among 130 teams.

Contribution

It introduces a novel application of Deep Q-Networks to AI Soccer with custom reward, state, and action spaces, demonstrating competitive performance.

Findings

Agents trained with DQN performed well in competition.

Achieved top 16 placement among 130 teams.

Demonstrated feasibility of reinforcement learning in multi-agent sports games.

Abstract

Reinforcement learning has shown an outstanding performance in the applications of games, particularly in Atari games as well as Go. Based on these successful examples, we attempt to apply one of the well-known reinforcement learning algorithms, Deep Q-Network, to the AI Soccer game. AI Soccer is a 5:5 robot soccer game where each participant develops an algorithm that controls five robots in a team to defeat the opponent participant. Deep Q-Network is designed to implement our original rewards, the state space, and the action space to train each agent so that it can take proper actions in different situations during the game. Our algorithm was able to successfully train the agents, and its performance was preliminarily proven through the mini-competition against 10 teams wishing to take part in the AI Soccer international competition. The competition was organized by the AI World Cup committee, in conjunction with the WCG 2019 Xi'an AI Masters. With our algorithm, we got the achievement of advancing to the round of 16 in this international competition with 130 teams from 39 countries.