Removing Skill Bias from Gaming Statistics

TL;DR

This paper identifies and addresses the skill bias in gaming statistics, proposing a method to accurately measure move value regardless of player skill level, which could improve reinforcement learning and game analysis.

Contribution

It introduces a simple toy model to quantify skill bias and a modular, generalizable method to remove this bias from gaming data, independent of player skill levels.

Findings

The method effectively removes skill bias from data.

Results are consistent across different player skill groups.

The approach is simple, modular, and data-efficient.

Abstract

"The chance to win given a certain move" is an easily obtainable quantity from data and often quoted in gaming statistics. It is also the fundamental quantity that reinforcement learning AI bases on. Unfortunately, this conditional probability can be misleading. Unless all players are equally skilled, this number does not tell us the intrinsic value of such move. That is because conditioning on one good move also inevitably selects a subset of better players. They tend to make other good moves, which also contribute to the extra winning chance. We present a simple toy model to quantify this "skill bias" effect, and then propose a general method to remove it. Our method is modular, generalizable, and also only requires easily obtainable quantities from data. In particular, it gets the same answer independent of whether the data comes from a group of good or bad players. This may help us to eventually break free from the conventional wisdom of "learning from the experts" and avoid the Group Thinking pitfall.