LOQA: Learning with Opponent Q-Learning Awareness

TL;DR

LOQA is a decentralized reinforcement learning algorithm designed to improve individual utility and cooperation among agents in general-sum games, demonstrating state-of-the-art results with low computational costs.

Contribution

LOQA introduces a novel opponent-aware Q-learning method that enhances multi-agent cooperation and efficiency in partially competitive environments.

Findings

Achieves state-of-the-art performance in benchmark games

Reduces computational complexity compared to existing methods

Effectively fosters cooperation among adversarial agents

Abstract

In various real-world scenarios, interactions among agents often resemble the dynamics of general-sum games, where each agent strives to optimize its own utility. Despite the ubiquitous relevance of such settings, decentralized machine learning algorithms have struggled to find equilibria that maximize individual utility while preserving social welfare. In this paper we introduce Learning with Opponent Q-Learning Awareness (LOQA), a novel, decentralized reinforcement learning algorithm tailored to optimizing an agent's individual utility while fostering cooperation among adversaries in partially competitive environments. LOQA assumes the opponent samples actions proportionally to their action-value function Q. Experimental results demonstrate the effectiveness of LOQA at achieving state-of-the-art performance in benchmark scenarios such as the Iterated Prisoner's Dilemma and the Coin Game. LOQA achieves these outcomes with a significantly reduced computational footprint, making it a promising approach for practical multi-agent applications.