Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning

TL;DR

This paper introduces an episodic future thinking mechanism for multi-agent reinforcement learning that enables agents to infer other agents' characters, predict their future actions, and adaptively choose optimal strategies, improving performance in diverse multi-agent scenarios.

Contribution

The paper proposes a novel EFT mechanism with a multi-character policy for character inference and future action prediction in multi-agent RL, inspired by cognitive processes.

Findings

EFT mechanism improves reward in multi-agent autonomous driving scenarios.

Accurate character inference enhances decision-making and performance.

Effectiveness persists across societies with varying character diversity.

Abstract

Understanding cognitive processes in multi-agent interactions is a primary goal in cognitive science. It can guide the direction of artificial intelligence (AI) research toward social decision-making in multi-agent systems, which includes uncertainty from character heterogeneity. In this paper, we introduce an episodic future thinking (EFT) mechanism for a reinforcement learning (RL) agent, inspired by cognitive processes observed in animals. To enable future thinking functionality, we first develop a multi-character policy that captures diverse characters with an ensemble of heterogeneous policies. Here, the character of an agent is defined as a different weight combination on reward components, representing distinct behavioral preferences. The future thinking agent collects observation-action trajectories of the target agents and uses the pre-trained multi-character policy to infer their characters. Once the character is inferred, the agent predicts the upcoming actions of target agents and simulates the potential future scenario. This capability allows the agent to adaptively select the optimal action, considering the predicted future scenario in multi-agent interactions. To evaluate the proposed mechanism, we consider the multi-agent autonomous driving scenario with diverse driving traits and multiple particle environments. Simulation results demonstrate that the EFT mechanism with accurate character inference leads to a higher reward than existing multi-agent solutions. We also confirm that the effect of reward improvement remains valid across societies with different levels of character diversity.