Towards AI-Powered Video Assistant Referee System (VARS) for Association Football

TL;DR

This paper introduces a semi-automated AI-powered VAR system for football that improves foul recognition and sanctioning accuracy, aiming to support referees at all levels and achieve human-level performance.

Contribution

The paper presents a novel semi-automated VAR system leveraging multi-view video analysis, setting new state-of-the-art results on the SoccerNet-MVFoul dataset.

Findings

Achieves 50% foul recognition accuracy

Achieves 46% sanctioning accuracy

Comparable to human performance in foul classification

Abstract

Over the past decade, the technology used by referees in football has improved substantially, enhancing the fairness and accuracy of decisions. This progress has culminated in the implementation of the Video Assistant Referee (VAR), an innovation that enables backstage referees to review incidents on the pitch from multiple points of view. However, the VAR is currently limited to professional leagues due to its expensive infrastructure and the lack of referees worldwide. In this paper, we present the semi-automated Video Assistant Referee System (VARS) that leverages the latest findings in multi-view video analysis. VARS sets a new state-of-the-art on the SoccerNet-MVFoul dataset, a multi-view video dataset of football fouls. Our VARS achieves a new state-of-the-art on the SoccerNet-MVFoul dataset by recognizing the type of foul in 50% of instances and the appropriate sanction in 46% of cases. Finally, we conducted a comparative study to investigate human performance in classifying fouls and their corresponding severity and compared these findings to our VARS. The results of our study highlight the potential of our VARS to reach human performance and support football refereeing across all levels of professional and amateur federations.