Multi-Issue Bargaining With Deep Reinforcement Learning

TL;DR

This paper explores the application of deep reinforcement learning to multi-issue bargaining, demonstrating that neural agents can learn to exploit, adapt, and cooperate, leading to fairer negotiation outcomes in complex bargaining scenarios.

Contribution

It introduces a novel approach using actor-critic networks for bargaining, showing how neural agents can learn to exploit, adapt, and cooperate in negotiation games, which departs from classical game theory.

Findings

Neural agents learn to exploit time-based agents and prefer Cauchy distribution for sampling offers.

Agents demonstrate adaptive behavior across different concession and discount strategies.

Neural agents can cooperate and use non-credible threats to achieve fairer outcomes.

Abstract

Negotiation is a process where agents aim to work through disputes and maximize their surplus. As the use of deep reinforcement learning in bargaining games is unexplored, this paper evaluates its ability to exploit, adapt, and cooperate to produce fair outcomes. Two actor-critic networks were trained for the bidding and acceptance strategy, against time-based agents, behavior-based agents, and through self-play. Gameplay against these agents reveals three key findings. 1) Neural agents learn to exploit time-based agents, achieving clear transitions in decision preference values. The Cauchy distribution emerges as suitable for sampling offers, due to its peaky center and heavy tails. The kurtosis and variance sensitivity of the probability distributions used for continuous control produce trade-offs in exploration and exploitation. 2) Neural agents demonstrate adaptive behavior against different combinations of concession, discount factors, and behavior-based strategies. 3) Most importantly, neural agents learn to cooperate with other behavior-based agents, in certain cases utilizing non-credible threats to force fairer results. This bears similarities with reputation-based strategies in the evolutionary dynamics, and departs from equilibria in classical game theory.