A Q-values Sharing Framework for Multiagent Reinforcement Learning under Budget Constraint

TL;DR

This paper introduces PSAF, a Q-values sharing framework for cooperative multiagent reinforcement learning that improves learning efficiency under communication budget constraints by enabling agents to decide when to ask for and share Q-values.

Contribution

The paper proposes a novel PSAF framework allowing agents to adaptively share and request Q-values, enhancing cooperation under limited communication budgets.

Findings

PSAF outperforms existing advising methods in various multiagent tasks.

Sharing Q-values improves learning efficiency and cooperation.

Adaptive advising reduces communication costs while maintaining performance.

Abstract

In teacher-student framework, a more experienced agent (teacher) helps accelerate the learning of another agent (student) by suggesting actions to take in certain states. In cooperative multiagent reinforcement learning (MARL), where agents need to cooperate with one another, a student may fail to cooperate well with others even by following the teachers' suggested actions, as the polices of all agents are ever changing before convergence. When the number of times that agents communicate with one another is limited (i.e., there is budget constraint), the advising strategy that uses actions as advices may not be good enough. We propose a partaker-sharer advising framework (PSAF) for cooperative MARL agents learning with budget constraint. In PSAF, each Q-learner can decide when to ask for Q-values and share its Q-values. We perform experiments in three typical multiagent learning problems. Evaluation results show that our approach PSAF outperforms existing advising methods under both unlimited and limited budget, and we give an analysis of the impact of advising actions and sharing Q-values on agents' learning.