Decision Market Based Learning For Multi-agent Contextual Bandit Problems

TL;DR

This paper explores decision markets as mechanisms for multi-agent learning in contextual bandit problems, demonstrating their effectiveness in aggregating distributed information and analyzing incentive compatibility issues through simulations.

Contribution

It introduces a multi-agent system utilizing decision markets and proper scoring rules to efficiently solve contextual bandit problems with distributed information.

Findings

Distributed multi-agent system performs as well as centralized approach.

Proper scoring rules enable agents to learn collaboratively.

Deterministic scoring rules lead to manipulative behaviors.

Abstract

Information is often stored in a distributed and proprietary form, and agents who own information are often self-interested and require incentives to reveal their information. Suitable mechanisms are required to elicit and aggregate such distributed information for decision making. In this paper, we use simulations to investigate the use of decision markets as mechanisms in a multi-agent learning system to aggregate distributed information for decision-making in a contextual bandit problem. The system utilises strictly proper decision scoring rules to assess the accuracy of probabilistic reports from agents, which allows agents to learn to solve the contextual bandit problem jointly. Our simulations show that our multi-agent system with distributed information can be trained as efficiently as a centralised counterpart with a single agent that receives all information. Moreover, we use our system to investigate scenarios with deterministic decision scoring rules which are not incentive compatible. We observe the emergence of more complex dynamics with manipulative behaviour, which agrees with existing theoretical analyses.