Preference Communication in Multi-Objective Normal-Form Games

TL;DR

This paper introduces novel preference communication protocols in multi-objective normal-form games, enabling agents to learn effective strategies and revealing how communication influences policy emergence and game dynamics.

Contribution

It proposes four new preference communication protocols inspired by Stackelberg games and studies their impact on agent learning and policy emergence in multi-objective games.

Findings

Preference communication can change the learning process.

Cyclic policies can emerge due to communication.

Communication naturally arises in various game settings.

Abstract

We consider preference communication in two-player multi-objective normal-form games. In such games, the payoffs resulting from joint actions are vector-valued. Taking a utility-based approach, we assume there exists a utility function for each player which maps vectors to scalar utilities and consider agents that aim to maximise the utility of expected payoff vectors. As agents typically do not know their opponent's utility function or strategy, they must learn policies to interact with each other. Inspired by Stackelberg games, we introduce four novel preference communication protocols to aid agents in arriving at adequate solutions. Each protocol describes a specific approach for one agent to communicate preferences over their actions and how another agent responds. Additionally, to study when communication emerges, we introduce a communication protocol where agents must learn when to communicate. These protocols are subsequently evaluated on a set of five benchmark games against baseline agents that do not communicate. We find that preference communication can alter the learning process and lead to the emergence of cyclic policies which had not been previously observed in this setting. We further observe that the resulting policies can heavily depend on the characteristics of the game that is played. Lastly, we find that communication naturally emerges in both cooperative and self-interested settings.