Towards optimized actions in critical situations of soccer games with deep reinforcement learning

TL;DR

This paper introduces a deep reinforcement learning approach with a new state representation to optimize decision-making in critical soccer game situations, demonstrating improved policies over traditional methods.

Contribution

It proposes a novel state representation and batch reinforcement learning framework to derive optimal actions in soccer, validated on real match data.

Findings

Optimized policies outperform behavior policies in all tested matches.

The learned policies align closely with real-world decision patterns.

Certain actions like fouls or ball out can be more rewarding than shots in specific contexts.

Abstract

Soccer is a sparse rewarding game: any smart or careless action in critical situations can change the result of the match. Therefore players, coaches, and scouts are all curious about the best action to be performed in critical situations, such as the times with a high probability of losing ball possession or scoring a goal. This work proposes a new state representation for the soccer game and a batch reinforcement learning to train a smart policy network. This network gets the contextual information of the situation and proposes the optimal action to maximize the expected goal for the team. We performed extensive numerical experiments on the soccer logs made by InStat for 104 European soccer matches. The results show that in all 104 games, the optimized policy obtains higher rewards than its counterpart in the behavior policy. Besides, our framework learns policies that are close to the expected behavior in the real world. For instance, in the optimized policy, we observe that some actions such as foul, or ball out can be sometimes more rewarding than a shot in specific situations.