Transfer in Reinforcement Learning via Regret Bounds for Learning Agents

TL;DR

This paper analyzes how sharing observations among multiple reinforcement learning agents can reduce total regret, providing theoretical bounds that quantify the benefit of transfer learning in multi-agent environments.

Contribution

It introduces regret bounds for multi-agent reinforcement learning, demonstrating the advantage of shared observations over individual learning in terms of regret reduction.

Findings

Sharing observations reduces total regret by a factor of sqrt(agents)

Regret bounds quantify the benefit of transfer in multi-agent RL

Theoretical analysis supports observation sharing in multi-agent transfer learning

Abstract

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $\aleph$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing her average reward. We show that when the agents share their observations the total regret of all agents is smaller by a factor of $\sqrt{\aleph}$ compared to the case when each agent has to rely on the information collected by herself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.