Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?

TL;DR

This paper investigates the limitations of training reinforcement learning agents to optimize score differences instead of win/lose outcomes in perfect information games, revealing empirical and theoretical insights into their suboptimality.

Contribution

It provides empirical evidence and a theoretical framework explaining why score-based training may lead to suboptimal policies in deterministic, perfect information games.

Findings

Score-based training often results in suboptimal policies.

Outcome-optimal policies prefer higher score variance in losing states.

Deterministic games can behave like nondeterministic ones under approximation.

Abstract

In the last years, the DeepMind algorithm AlphaZero has become the state of the art to efficiently tackle perfect information two-player zero-sum games with a win/lose outcome. However, when the win/lose outcome is decided by a final score difference, AlphaZero may play score-suboptimal moves because all winning final positions are equivalent from the win/lose outcome perspective. This can be an issue, for instance when used for teaching, or when trying to understand whether there is a better move. Moreover, there is the theoretical quest for the perfect game. A naive approach would be training an AlphaZero-like agent to predict score differences instead of win/lose outcomes. Since the game of Go is deterministic, this should as well produce an outcome-optimal play. However, it is a folklore belief that "this does not work". In this paper, we first provide empirical evidence for this belief. We then give a theoretical interpretation of this suboptimality in general perfect information two-player zero-sum game where the complexity of a game like Go is replaced by the randomness of the environment. We show that an outcome-optimal policy has a different preference for uncertainty when it is winning or losing. In particular, when in a losing state, an outcome-optimal agent chooses actions leading to a higher score variance. We then posit that when approximation is involved, a deterministic game behaves like a nondeterministic game, where the score variance is modeled by how uncertain the position is. We validate this hypothesis in AlphaZero-like software with a human expert.