OpenHoldem: A Benchmark for Large-Scale Imperfect-Information Game Research

TL;DR

OpenHoldem is a comprehensive toolkit that provides standardized evaluation, strong baselines, and an online platform to advance research in large-scale imperfect-information games like No-limit Texas Hold'em.

Contribution

It introduces a standardized evaluation protocol, four strong baseline AIs, and an accessible online testing platform for NLTH research.

Findings

Facilitates fair comparison of NLTH AIs

Provides publicly available strong baseline models

Enables easy online evaluation and benchmarking

Abstract

Owning to the unremitting efforts by a few institutes, significant progress has recently been made in designing superhuman AIs in No-limit Texas Hold'em (NLTH), the primary testbed for large-scale imperfect-information game research. However, it remains challenging for new researchers to study this problem since there are no standard benchmarks for comparing with existing methods, which seriously hinders further developments in this research area. In this work, we present OpenHoldem, an integrated toolkit for large-scale imperfect-information game research using NLTH. OpenHoldem makes three main contributions to this research direction: 1) a standardized evaluation protocol for thoroughly evaluating different NLTH AIs, 2) four publicly available strong baselines for NLTH AI, and 3) an online testing platform with easy-to-use APIs for public NLTH AI evaluation. We have released OpenHoldem at holdem.ia.ac.cn, hoping it facilitates further studies on the unsolved theoretical and computational issues in this area and cultivate crucial research problems like opponent modeling and human-computer interactive learning.