Adapting Skill Ratings to Luck-Based Hidden-Information Games

TL;DR

This paper introduces a modified Elo rating system tailored for luck-based, incomplete-information games like Rummy, incorporating score metrics and initial hand quality to better evaluate player skill amidst randomness.

Contribution

It proposes a novel Elo framework that accounts for luck and initial conditions, improving skill assessment in stochastic, incomplete-information games.

Findings

Achieves stable convergence in skill ratings.

Outperforms traditional Elo in predictive accuracy.

Effectively captures skill, strategy, and randomness interplay.

Abstract

Rating systems play a crucial role in evaluating player skill across competitive environments. The Elo rating system, originally designed for deterministic and information-complete games such as chess, has been widely adopted and modified in various domains. However, the traditional Elo rating system only considers game outcomes for rating calculation and assumes uniform initial states across players. This raises important methodological challenges in skill modelling for popular partially randomized incomplete-information games such as Rummy. In this paper, we examine the limitations of conventional Elo ratings when applied to luck-driven environments and propose a modified Elo framework specifically tailored for Rummy. Our approach incorporates score-based performance metrics and explicitly models the influence of initial hand quality to disentangle skill from luck. Through extensive simulations involving 270,000 games across six strategies of varying sophistication, we demonstrate that our proposed system achieves stable convergence, superior discriminative power, and enhanced predictive accuracy compared to traditional Elo formulations. The framework maintains computational simplicity while effectively capturing the interplay of skill, strategy, and randomness, with broad applicability to other stochastic competitive environments.