Hierarchical RNNs-Based Transformers MADDPG for Mixed Cooperative-Competitive Environments

TL;DR

This paper introduces HRTMADDPG, a hierarchical transformer-based multi-agent reinforcement learning method that combines RNNs and transformers to better capture temporal and inter-sequence relationships in cooperative-competitive environments.

Contribution

It proposes a novel hierarchical model integrating RNNs and transformers for multi-agent reinforcement learning, enhancing the understanding of temporal and inter-sequence dependencies.

Findings

Improved learning efficiency in multi-agent environments.

Effective encoding of multi-step temporal information.

Enhanced correlation learning between sequences.

Abstract

At present, attention mechanism has been widely applied to the fields of deep learning models. Structural models that based on attention mechanism can not only record the relationships between features position, but also can measure the importance of different features based on their weights. By establishing dynamically weighted parameters for choosing relevant and irrelevant features, the key information can be strengthened, and the irrelevant information can be weakened. Therefore, the efficiency of deep learning algorithms can be significantly elevated and improved. Although transformers have been performed very well in many fields including reinforcement learning, there are still many problems and applications can be solved and made with transformers within this area. MARL (known as Multi-Agent Reinforcement Learning) can be recognized as a set of independent agents trying to adapt and learn through their way to reach the goal. In order to emphasize the relationship between each MDP decision in a certain time period, we applied the hierarchical coding method and validated the effectiveness of this method. This paper proposed a hierarchical transformers MADDPG based on RNN which we call it Hierarchical RNNs-Based Transformers MADDPG(HRTMADDPG). It consists of a lower level encoder based on RNNs that encodes multiple step sizes in each time sequence, and it also consists of an upper sequence level encoder based on transformer for learning the correlations between multiple sequences so that we can capture the causal relationship between sub-time sequences and make HRTMADDPG more efficient.