Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

TL;DR

This paper introduces a novel method combining counterfactual regret minimization with information gain to efficiently solve extensive games in unknown environments, enhancing exploration and NE approximation accuracy.

Contribution

It proposes a curiosity-driven approach that integrates information gain into CFR, enabling effective NE computation in uncertain environments with fewer interactions.

Findings

Reduces environment interactions compared to baselines.

Achieves more accurate NE approximation.

Demonstrates effectiveness on Kuhn and Leduc poker.

Abstract

Counterfactual regret minimization (CFR) is an effective algorithm for solving extensive games with imperfect information (IIEGs). However, CFR is only allowed to be applied in known environments, where the transition function of the chance player and the reward function of the terminal node in IIEGs are known. In uncertain situations, such as reinforcement learning (RL) problems, CFR is not applicable. Thus, applying CFR in unknown environments is a significant challenge that can also address some difficulties in the real world. Currently, advanced solutions require more interactions with the environment and are limited by large single-sampling variances to narrow the gap with the real environment. In this paper, we propose a method that combines CFR with information gain to compute the Nash equilibrium (NE) of IIEGs with unknown environments. We use a curiosity-driven approach to explore unknown environments and minimize the discrepancy between uncertain and real environments. Additionally, by incorporating information into the reward, the average strategy calculated by CFR can be directly implemented as the interaction policy with the environment, thereby improving the exploration efficiency of our method in uncertain environments. Through experiments on standard testbeds such as Kuhn poker and Leduc poker, our method significantly reduces the number of interactions with the environment compared to the different baselines and computes a more accurate approximate NE within the same number of interaction rounds.