Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces an adaptive context length optimization framework for multi-agent reinforcement learning, utilizing low-frequency truncation and temporal gradient analysis to improve exploration and convergence in complex environments.

Contribution

It proposes a novel MARL framework with dynamic context length optimization using a central agent and Fourier-based low-frequency truncation for better environment representation.

Findings

Achieves state-of-the-art performance on long-term dependency tasks.

Enhances exploration efficiency and convergence in MARL environments.

Effectively filters redundant information through low-frequency truncation.

Abstract

Recently, deep multi-agent reinforcement learning (MARL) has demonstrated promising performance for solving challenging tasks, such as long-term dependencies and non-Markovian environments. Its success is partly attributed to conditioning policies on large fixed context length. However, such large fixed context lengths may lead to limited exploration efficiency and redundant information. In this paper, we propose a novel MARL framework to obtain adaptive and effective contextual information. Specifically, we design a central agent that dynamically optimizes context length via temporal gradient analysis, enhancing exploration to facilitate convergence to global optima in MARL. Furthermore, to enhance the adaptive optimization capability of the context length, we present an efficient input representation for the central agent, which effectively filters redundant information. By leveraging a Fourier-based low-frequency truncation method, we extract global temporal trends across decentralized agents, providing an effective and efficient representation of the MARL environment. Extensive experiments demonstrate that the proposed method achieves state-of-the-art (SOTA) performance on long-term dependency tasks, including PettingZoo, MiniGrid, Google Research Football (GRF), and StarCraft Multi-Agent Challenge v2 (SMACv2).