Reinforcing the World's Edge: A Continual Learning Problem in the Multi-Agent-World Boundary

TL;DR

This paper investigates how the agent-world boundary affects decision structure transfer in reinforcement learning, especially in multi-agent settings, highlighting the challenges of boundary drift and its impact on invariant cores.

Contribution

It introduces a formal analysis of the agent-world boundary in decentralized MARL and demonstrates how boundary drift causes the loss of invariant decision cores across episodes.

Findings

Invariant cores can be constructed in stationary MDPs

Boundary drift causes the invariant core to shrink or vanish

Policy updates induce non-stationarity affecting decision invariants

Abstract

Reusable decision structure survives across episodes in reinforcement learning, but this depends on how the agent--world boundary is drawn. In stationary, finite-horizon MDPs, an invariant core: the (not-necessarily contiguous) subsequences of state--action pairs shared by all successful trajectories (optionally under a simple abstraction) can be constructed. Under mild goal-conditioned assumptions, it's existence can be proven and explained by how the core captures prototypes that transfer across episodes. When the same task is embedded in a decentralized Markov game and the peer agent is folded into the world, each peer-policy update induces a new MDP; the per-episode invariant core can shrink or vanish, even with small changes to the induced world dynamics, sometimes leaving only the individual task core or just nothing. This policy-induced non-stationarity can be quantified with a variation budget over the induced kernels and rewards, linking boundary drift to loss of invariants. The view that a continual RL problem arises from instability of the agent--world boundary (rather than exogenous task switches) in decentralized MARL suggests future work on preserving, predicting, or otherwise managing boundary drift.