Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data

TL;DR

This paper introduces a new evaluation metric called Balance score for probability estimation models, especially in esports, addressing limitations of existing metrics like accuracy, Brier score, and ECE, and demonstrating its effectiveness through simulations and real data.

Contribution

It proposes the Balance score as a novel, effective metric for evaluating probability estimation models, improving upon traditional metrics in esports and general applications.

Findings

Balance score effectively approximates true calibration error.

Extensive evaluations show the metric's robustness and applicability.

Proposed metric outperforms traditional metrics in reliability assessment.

Abstract

Probability estimation models play an important role in various fields, such as weather forecasting, recommendation systems, and sports analysis. Among several models estimating probabilities, it is difficult to evaluate which model gives reliable probabilities since the ground-truth probabilities are not available. The win probability estimation model for esports, which calculates the win probability under a certain game state, is also one of the fields being actively studied in probability estimation. However, most of the previous works evaluated their models using accuracy, a metric that only can measure the performance of discrimination. In this work, we firstly investigate the Brier score and the Expected Calibration Error (ECE) as a replacement of accuracy used as a performance evaluation metric for win probability estimation models in esports field. Based on the analysis, we propose a novel metric called Balance score which is a simple yet effective metric in terms of six good properties that probability estimation metric should have. Under the general condition, we also found that the Balance score can be an effective approximation of the true expected calibration error which has been imperfectly approximated by ECE using the binning technique. Extensive evaluations using simulation studies and real game snapshot data demonstrate the promising potential to adopt the proposed metric not only for the win probability estimation model for esports but also for evaluating general probability estimation models.