Context-Based Concurrent Experience Sharing in Multiagent Systems

TL;DR

This paper presents a scalable, context-based experience sharing method for multi-agent systems that enhances learning efficiency and performance in large, dynamic environments through adaptive, low-overhead communication.

Contribution

It introduces an online, distributed, supervisor-directed technique for identifying contexts and facilitating experience sharing among agents, improving scalability and robustness.

Findings

Significant performance improvements in large-scale distributed tasks.

Robustness to noise and suboptimal context features.

Communication costs scale linearly with supervisor-to-agent ratio.

Abstract

One of the key challenges for multi-agent learning is scalability. In this paper, we introduce a technique for speeding up multi-agent learning by exploiting concurrent and incremental experience sharing. This solution adaptively identifies opportunities to transfer experiences between agents and allows for the rapid acquisition of appropriate policies in large-scale, stochastic, homogeneous multi-agent systems. We introduce an online, distributed, supervisor-directed transfer technique for constructing high-level characterizations of an agent's dynamic learning environment---called contexts---which are used to identify groups of agents operating under approximately similar dynamics within a short temporal window. A set of supervisory agents computes contextual information for groups of subordinate agents, thereby identifying candidates for experience sharing. Our method uses a tiered architecture to propagate, with low communication overhead, state, action, and reward data amongst the members of each dynamically-identified information-sharing group. We applied this method to a large-scale distributed task allocation problem with hundreds of information-sharing agents operating in an unknown, non-stationary environment. We demonstrate that our approach results in significant performance gains, that it is robust to noise-corrupted or suboptimal context features, and that communication costs scale linearly with the supervisor-to-subordinate ratio.