Quantifying the Burden of Exploration and the Unfairness of Free Riding

TL;DR

This paper investigates how multiple decision makers in multi-armed bandit problems can exploit each other's actions to achieve low regret, revealing conditions under which free riding is highly effective in both stochastic and linear contextual settings.

Contribution

It introduces a formal analysis of free riding in multi-agent bandit scenarios, demonstrating that observing other agents' actions can drastically reduce regret under standard assumptions.

Findings

Free riders can achieve $O(1)$ regret by observing self-reliant agents.

Conditions for low regret include arms being pulled logarithmically often and agents using zero-regret algorithms.

In linear contextual bandits, free riders succeed when their context is a small combination of others' contexts.

Abstract

We consider the multi-armed bandit setting with a twist. Rather than having just one decision maker deciding which arm to pull in each round, we have $n$ different decision makers (agents). In the simple stochastic setting, we show that a "free-riding" agent observing another "self-reliant" agent can achieve just $O(1)$ regret, as opposed to the regret lower bound of $\Omega (\log t)$ when one decision maker is playing in isolation. This result holds whenever the self-reliant agent's strategy satisfies either one of two assumptions: (1) each arm is pulled at least $\gamma \ln t$ times in expectation for a constant $\gamma$ that we compute, or (2) the self-reliant agent achieves $o(t)$ realized regret with high probability. Both of these assumptions are satisfied by standard zero-regret algorithms. Under the second assumption, we further show that the free rider only needs to observe the number of times each arm is pulled by the self-reliant agent, and not the rewards realized. In the linear contextual setting, each arm has a distribution over parameter vectors, each agent has a context vector, and the reward realized when an agent pulls an arm is the inner product of that agent's context vector with a parameter vector sampled from the pulled arm's distribution. We show that the free rider can achieve $O(1)$ regret in this setting whenever the free rider's context is a small (in $L_2$-norm) linear combination of other agents' contexts and all other agents pull each arm $\Omega (\log t)$ times with high probability. Again, this condition on the self-reliant players is satisfied by standard zero-regret algorithms like UCB. We also prove a number of lower bounds.