Leveling the Playing Field -- Fairness in AI Versus Human Game Benchmarks

TL;DR

This paper reviews the fairness of AI versus human performance in game benchmarks, analyzing media and academic claims to understand the true nature of AI achievements in gaming.

Contribution

It provides a critical analysis of claims about AI's dominance in game benchmarks and discusses factors influencing perceptions of fairness between humans and AI.

Findings

Media often exaggerate AI performance in games

Factors affecting fairness perceptions are analyzed

The paper highlights discrepancies between claims and actual results

Abstract

From the beginning if the history of AI, there has been interest in games as a platform of research. As the field developed, human-level competence in complex games became a target researchers worked to reach. Only relatively recently has this target been finally met for traditional tabletop games such as Backgammon, Chess and Go. Current research focus has shifted to electronic games, which provide unique challenges. As is often the case with AI research, these results are liable to be exaggerated or misrepresented by either authors or third parties. The extent to which these games benchmark consist of fair competition between human and AI is also a matter of debate. In this work, we review the statements made by authors and third parties in the general media and academic circle about these game benchmark results and discuss factors that can impact the perception of fairness in the contest between humans and machines