Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces EF-OnTL, a novel transfer learning algorithm for multi-agent reinforcement learning that operates without experts, dynamically selecting source agents based on performance and uncertainty, and demonstrates its effectiveness across various benchmark tasks.

Contribution

The paper proposes EF-OnTL, an expert-free, real-time transfer learning method with a new uncertainty estimation technique, improving multi-agent RL without external knowledge or expert agents.

Findings

EF-OnTL performs comparably to advice-based methods without external input.

EF-OnTL outperforms no-transfer approaches, especially in complex tasks.

The proposed sars-RND improves uncertainty estimation in RL environments.

Abstract

Transfer learning in Reinforcement Learning (RL) has been widely studied to overcome training issues of Deep-RL, i.e., exploration cost, data availability and convergence time, by introducing a way to enhance training phase with external knowledge. Generally, knowledge is transferred from expert-agents to novices. While this fixes the issue for a novice agent, a good understanding of the task on expert agent is required for such transfer to be effective. As an alternative, in this paper we propose Expert-Free Online Transfer Learning (EF-OnTL), an algorithm that enables expert-free real-time dynamic transfer learning in multi-agent system. No dedicated expert exists, and transfer source agent and knowledge to be transferred are dynamically selected at each transfer step based on agents' performance and uncertainty. To improve uncertainty estimation, we also propose State Action Reward Next-State Random Network Distillation (sars-RND), an extension of RND that estimates uncertainty from RL agent-environment interaction. We demonstrate EF-OnTL effectiveness against a no-transfer scenario and advice-based baselines, with and without expert agents, in three benchmark tasks: Cart-Pole, a grid-based Multi-Team Predator-Prey (mt-pp) and Half Field Offense (HFO). Our results show that EF-OnTL achieve overall comparable performance when compared against advice-based baselines while not requiring any external input nor threshold tuning. EF-OnTL outperforms no-transfer with an improvement related to the complexity of the task addressed.