The Partially Observable History Process

TL;DR

The paper introduces the partially observable history process (POHP) formalism, unifying various reinforcement learning models and enabling cross-domain algorithm design and theoretical analysis.

Contribution

It presents a new formalism that unifies multiple RL models, simplifying analysis and algorithm development across single and multi-agent settings.

Findings

Unifies MDPs, Markov games, and extensive-form games under POHP.

Provides a streamlined interface for designing RL algorithms.

Generalizes the extensive-form regret minimization algorithm.

Abstract

We introduce the partially observable history process (POHP) formalism for reinforcement learning. POHP centers around the actions and observations of a single agent and abstracts away the presence of other players without reducing them to stochastic processes. Our formalism provides a streamlined interface for designing algorithms that defy categorization as exclusively single or multi-agent, and for developing theory that applies across these domains. We show how the POHP formalism unifies traditional models including the Markov decision process, the Markov game, the extensive-form game, and their partially observable extensions, without introducing burdensome technical machinery or violating the philosophical underpinnings of reinforcement learning. We illustrate the utility of our formalism by concisely exploring observable sequential rationality, examining some theoretical properties of general immediate regret minimization, and generalizing the extensive-form regret minimization (EFR) algorithm.