Counter-Factual Reinforcement Learning: How to Model Decision-Makers That Anticipate The Future

TL;DR

This paper presents a new framework combining game theory and reinforcement learning to model multi-stage decision-making by bounded rational humans, enabling strategic anticipation and computational efficiency in complex scenarios.

Contribution

It introduces the iterated semi network-form game framework and level-K reinforcement learning, extending bounded rational models to multi-stage games with practical algorithms.

Findings

Effective modeling of human strategic behavior in cyber battles

Decomposition into smaller RL problems enhances computational tractability

Predicted behaviors align with real human defender and attacker actions

Abstract

This paper introduces a novel framework for modeling interacting humans in a multi-stage game. This "iterated semi network-form game" framework has the following desirable characteristics: (1) Bounded rational players, (2) strategic players (i.e., players account for one another's reward functions when predicting one another's behavior), and (3) computational tractability even on real-world systems. We achieve these benefits by combining concepts from game theory and reinforcement learning. To be precise, we extend the bounded rational "level-K reasoning" model to apply to games over multiple stages. Our extension allows the decomposition of the overall modeling problem into a series of smaller ones, each of which can be solved by standard reinforcement learning algorithms. We call this hybrid approach "level-K reinforcement learning". We investigate these ideas in a cyber battle scenario over a smart power grid and discuss the relationship between the behavior predicted by our model and what one might expect of real human defenders and attackers.