MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation

TL;DR

MAGE introduces a meta-reinforcement learning framework for language agents that enhances their ability to strategically explore and exploit in multi-agent environments, leading to improved adaptability and generalization.

Contribution

The paper presents MAGE, a novel meta-RL approach for LLMs that incorporates multi-episode training, reflection, and population-based techniques to improve strategic exploration and exploitation.

Findings

MAGE outperforms baselines in exploration and exploitation tasks.

MAGE generalizes well to unseen opponents.

The framework enhances long-term adaptability of LLM agents.

Abstract

Large Language Model (LLM) agents have demonstrated remarkable proficiency in learned tasks, yet they often struggle to adapt to non-stationary environments with feedback. While In-Context Learning and external memory offer some flexibility, they fail to internalize the adaptive ability required for long-term improvement. Meta-Reinforcement Learning (meta-RL) provides an alternative by embedding the learning process directly within the model. However, existing meta-RL approaches for LLMs focus primarily on exploration in single-agent settings, neglecting the strategic exploitation necessary for multi-agent environments. We propose MAGE, a meta-RL framework that empowers LLM agents for strategic exploration and exploitation. MAGE utilizes a multi-episode training regime where interaction histories and reflections are integrated into the context window. By using the final episode reward as the objective, MAGE incentivizes the agent to refine its strategy based on past experiences. We further combine population-based training with an agent-specific advantage normalization technique to enrich agent diversity and ensure stable learning. Experiment results show that MAGE outperforms existing baselines in both exploration and exploitation tasks. Furthermore, MAGE exhibits strong generalization to unseen opponents, suggesting it has internalized the ability for strategic exploration and exploitation. Code is available at https://github.com/Lu-Yang666/MAGE.