Analysing Congestion Problems in Multi-agent Reinforcement Learning

TL;DR

This paper examines congestion issues in multi-agent reinforcement learning, analyzing existing methods like resource abstraction and difference rewards, introduces a new network-based congestion benchmark, and evaluates their effectiveness and limitations.

Contribution

It introduces the Road Network Domain as a new benchmark and compares MARL methods, revealing limitations of resource abstraction and advantages of difference rewards.

Findings

Difference rewards better capture environment dynamics.

Resource abstraction's effectiveness depends on undocumented assumptions.

The new RND benchmark highlights limitations of resource abstraction.

Abstract

Congestion problems are omnipresent in today's complex networks and represent a challenge in many research domains. In the context of Multi-agent Reinforcement Learning (MARL), approaches like difference rewards and resource abstraction have shown promising results in tackling such problems. Resource abstraction was shown to be an ideal candidate for solving large-scale resource allocation problems in a fully decentralized manner. However, its performance and applicability strongly depends on some, until now, undocumented assumptions. Two of the main congestion benchmark problems considered in the literature are: the Beach Problem Domain and the Traffic Lane Domain. In both settings the highest system utility is achieved when overcrowding one resource and keeping the rest at optimum capacity. We analyse how abstract grouping can promote this behaviour and how feasible it is to apply this approach in a real-world domain (i.e., what assumptions need to be satisfied and what knowledge is necessary). We introduce a new test problem, the Road Network Domain (RND), where the resources are no longer independent, but rather part of a network (e.g., road network), thus choosing one path will also impact the load on other paths having common road segments. We demonstrate the application of state-of-the-art MARL methods for this new congestion model and analyse their performance. RND allows us to highlight an important limitation of resource abstraction and show that the difference rewards approach manages to better capture and inform the agents about the dynamics of the environment.