Causal Knowledge Transfer for Multi-Agent Reinforcement Learning in Dynamic Environments

TL;DR

This paper presents a causal knowledge transfer framework in multi-agent reinforcement learning that enables agents to adapt to dynamic environments by sharing causal representations, reducing retraining needs and improving coordination.

Contribution

It introduces a novel causal knowledge transfer method allowing agents to share compact causal representations for adaptive recovery in non-stationary environments.

Findings

Agents with heterogeneous goals bridged about half the gap to fully retrained policies.

Causal knowledge transfer effectiveness depends on environment complexity and agent goals.

Zero-shot transfer of recovery strategies improved adaptation without retraining.

Abstract

[Context] Multi-agent reinforcement learning (MARL) has achieved notable success in environments where agents must learn coordinated behaviors. However, transferring knowledge across agents remains challenging in non-stationary environments with changing goals. [Problem] Traditional knowledge transfer methods in MARL struggle to generalize, and agents often require costly retraining to adapt. [Approach] This paper introduces a causal knowledge transfer framework that enables RL agents to learn and share compact causal representations of paths within a non-stationary environment. As the environment changes (new obstacles), agents' collisions require adaptive recovery strategies. We model each collision as a causal intervention instantiated as a sequence of recovery actions (a macro) whose effect corresponds to a causal knowledge of how to circumvent the obstacle while increasing the chances of achieving the agent's goal (maximizing cumulative reward). This recovery action macro is transferred online from a second agent and is applied in a zero-shot fashion, i.e., without retraining, just by querying a lookup model with local context information (collisions). [Results] Our findings reveal two key insights: (1) agents with heterogeneous goals were able to bridge about half of the gap between random exploration and a fully retrained policy when adapting to new environments, and (2) the impact of causal knowledge transfer depends on the interplay between environment complexity and agents' heterogeneous goals.