Evaluating one-shot tournament predictions

TL;DR

This paper introduces the Tournament Rank Probability Score (TRPS), a new flexible metric for evaluating pre-tournament predictions, and demonstrates how to optimally combine historical data into ensemble predictions to improve accuracy.

Contribution

The paper presents the TRPS as a novel evaluation metric and proposes a method to combine historical tournament predictions into optimal ensembles.

Findings

TRPS effectively evaluates partial and full tournament predictions.

Weighted TRPS allows emphasizing specific features of predictions.

Ensemble methods using historical data improve prediction accuracy.

Abstract

We introduce the Tournament Rank Probability Score (TRPS) as a measure to evaluate and compare pre-tournament predictions, where predictions of the full tournament results are required to be available before the tournament begins. The TRPS handles partial ranking of teams, gives credit to predictions that are only slightly wrong, and can be modified with weights to stress the importance of particular features of the tournament prediction. Thus, the Tournament Rank Prediction Score is more flexible than the commonly preferred log loss score for such tasks. In addition, we show how predictions from historic tournaments can be optimally combined into ensemble predictions in order to maximize the TRPS for a new tournament.