Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

TL;DR

This paper introduces CFCQL, a novel offline multi-agent reinforcement learning algorithm that applies conservative value estimation per agent in a counterfactual manner, improving performance and theoretical guarantees in high-dimensional multi-agent settings.

Contribution

CFCQL is the first method to perform agent-wise conservative regularization in a counterfactual way, with performance guarantees independent of the number of agents.

Findings

CFCQL outperforms existing methods on most datasets.

It maintains theoretical properties similar to single-agent conservative methods.

Demonstrates effectiveness in both discrete and continuous action environments.

Abstract

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. Tomitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.