Modeling evidential cooperation in large worlds

TL;DR

This paper develops a game-theoretic model of evidential cooperation in large worlds, exploring how agents with uncertain values can benefit from cooperation, and analyzes equilibrium concepts and stability issues.

Contribution

It introduces a formal model of ECL as an incomplete information bargaining problem, incorporating dependency equilibria and analyzing the relevance of the Nash bargaining solution.

Findings

All cooperators should maximize the same weighted utility sum for Pareto optimality.

The Nash bargaining solution can serve as a Schelling point in ECL.

Gains from trade decrease as belief in other agents diminishes.

Abstract

Evidential cooperation in large worlds (ECL) refers to the idea that humans and other agents can benefit by cooperating with similar agents with differing values in causally disconnected parts of a large universe. Cooperating provides agents with evidence that other similar agents are likely to cooperate too, resulting in gains from trade for all. This could be a crucial consideration for altruists. I develop a game-theoretic model of ECL as an incomplete information bargaining problem. The model incorporates uncertainty about others' value systems and empirical situations, and addresses the problem of selecting a compromise outcome. Using the model, I investigate issues with ECL and outline open technical and philosophical questions. I show that all cooperators must maximize the same weighted sum of utility functions to reach a Pareto optimal outcome. However, I argue against selecting a compromise outcome implicitly by normalizing utility functions. I review bargaining theory and argue that the Nash bargaining solution could be a relevant Schelling point. I introduce dependency equilibria (Spohn 2007), an equilibrium concept suitable for ECL, and generalize a folk theorem showing that the Nash bargaining solution is a dependency equilibrium. I discuss gains from trade given uncertain beliefs about other agents and analyze how these gains decrease in several toy examples as the belief in another agent decreases. Finally, I discuss open issues in my model. First, the Nash bargaining solution is sometimes not coalitionally stable, meaning that a subset of cooperators can unilaterally improve payoffs by deviating from the compromise. I investigate conditions under which stable payoff vectors exist. Second, I discuss how to model agents' default actions without ECL.