Learning Conventions in Multiagent Stochastic Domains using Likelihood Estimates

TL;DR

This paper explores learning coordinated strategies in multiagent stochastic domains with unobservable actions, using likelihood estimates and maximum likelihood methods to improve convergence to equilibrium.

Contribution

It introduces likelihood-based generalizations of fictitious play and proposes maximum likelihood for strategy elimination in partially observable multiagent systems.

Findings

Likelihood estimates affect convergence rates.

Maximum likelihood helps eliminate suboptimal strategies.

Approach facilitates equilibrium convergence in unobservable settings.

Abstract

Fully cooperative multiagent systems - those in which agents share a joint utility model- is of special interest in AI. A key problem is that of ensuring that the actions of individual agents are coordinated, especially in settings where the agents are autonomous decision makers. We investigate approaches to learning coordinated strategies in stochastic domains where an agent's actions are not directly observable by others. Much recent work in game theory has adopted a Bayesian learning perspective to the more general problem of equilibrium selection, but tends to assume that actions can be observed. We discuss the special problems that arise when actions are not observable, including effects on rates of convergence, and the effect of action failure probabilities and asymmetries. We also use likelihood estimates as a means of generalizing fictitious play learning models in our setting. Finally, we propose the use of maximum likelihood as a means of removing strategies from consideration, with the aim of convergence to a conventional equilibrium, at which point learning and deliberation can cease.