TVDO: Tchebycheff Value-Decomposition Optimization for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces TVDO, a novel value-decomposition method for multi-agent reinforcement learning that ensures policy consistency and outperforms existing methods on benchmark tasks.

Contribution

The paper proposes a Tchebycheff-based value decomposition approach that guarantees the sufficiency and necessity of IGM, improving policy consistency in cooperative MARL.

Findings

TVDO guarantees policy consistency in cooperative MARL.

TVDO outperforms state-of-the-art baselines on SMAC benchmark.

Theoretical proof of IGM satisfaction with Tchebycheff aggregation.

Abstract

In cooperative multiagent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) has recently attracted more attention due to the physical demand. However, the most dilemma therein is the inconsistency between jointly-trained policies and individually-executed actions. In this article, we propose a factorized Tchebycheff value-decomposition optimization (TVDO) method to overcome the trouble of inconsistency. In particular, a nonlinear Tchebycheff aggregation function is formulated to realize the global optimum by tightly constraining the upper bound of individual action-value bias, which is inspired by the Tchebycheff method of multi-objective optimization. We theoretically prove that, under no extra limitations, the factorized value decomposition with Tchebycheff aggregation satisfies the sufficiency and necessity of Individual-Global-Max (IGM), which guarantees the consistency between the global and individual optimal action-value function. Empirically, in the climb and penalty game, we verify that TVDO precisely expresses the global-to-individual value decomposition with a guarantee of policy consistency. Meanwhile, we evaluate TVDO in the SMAC benchmark, and extensive experiments demonstrate that TVDO achieves a significant performance superiority over some SOTA MARL baselines.