Leveraging Statistical Multi-Agent Online Planning with Emergent Value Function Approximation

TL;DR

This paper introduces EVADE, a novel method that enhances multi-agent online planning in stochastic environments by integrating emergent system behavior through reinforcement learning-based value function approximation, improving efficiency and performance.

Contribution

EVADE is the first approach to incorporate global emergent behavior into local online planning for multi-agent systems using reinforcement learning.

Findings

EVADE improves planning performance in complex stochastic environments.

EVADE enhances efficiency in planning breadth and depth.

EVADE outperforms baseline algorithms in a smart factory simulation.

Abstract

Making decisions is a great challenge in distributed autonomous environments due to enormous state spaces and uncertainty. Many online planning algorithms rely on statistical sampling to avoid searching the whole state space, while still being able to make acceptable decisions. However, planning often has to be performed under strict computational constraints making online planning in multi-agent systems highly limited, which could lead to poor system performance, especially in stochastic domains. In this paper, we propose Emergent Value function Approximation for Distributed Environments (EVADE), an approach to integrate global experience into multi-agent online planning in stochastic domains to consider global effects during local planning. For this purpose, a value function is approximated online based on the emergent system behaviour by using methods of reinforcement learning. We empirically evaluated EVADE with two statistical multi-agent online planning algorithms in a highly complex and stochastic smart factory environment, where multiple agents need to process various items at a shared set of machines. Our experiments show that EVADE can effectively improve the performance of multi-agent online planning while offering efficiency w.r.t. the breadth and depth of the planning process.