Heterogeneous Stochastic Interactions for Multiple Agents in a Multi-armed Bandit Problem

TL;DR

This paper studies a multi-agent multi-armed bandit problem where agents observe neighbors' choices and rewards in a stochastic network, proposing algorithms with performance bounds based on network structure and agent sociability.

Contribution

It introduces a novel multi-agent bandit model with heterogeneous stochastic interactions and provides algorithms with theoretical performance guarantees.

Findings

Performance bounds depend on network structure and agent sociability.

Predicted agent ranking aligns with actual performance both analytically and computationally.

The model captures complex social interactions influencing decision-making.

Abstract

We define and analyze a multi-agent multi-armed bandit problem in which decision-making agents can observe the choices and rewards of their neighbors. Neighbors are defined by a network graph with heterogeneous and stochastic interconnections. These interactions are determined by the sociability of each agent, which corresponds to the probability that the agent observes its neighbors. We design an algorithm for each agent to maximize its own expected cumulative reward and prove performance bounds that depend on the sociability of the agents and the network structure. We use the bounds to predict the rank ordering of agents according to their performance and verify the accuracy analytically and computationally.

Figures

Equations

$$N_i^k(T)=\sum _{t=1}^T\sum _{j=1}^K{\mathbb{I}}_{\{{\phi }_j^t=i\}}{\mathbb{I}}_{\{k,j\}}^t.$$

$${\hat{\mu }}_i^k(T)=\frac{S_i^k(T)}{N_i^k(T)},$$

$$\displaystyle\mathbb{I}_{\{\varphi_{k}^{t+1}=i\}}=\left\{\begin{array}[]{cl}1&,\>\>\>Q^{k}_{i}(t)=\max\{Q^{k}_{1}(t),\cdots,Q^{k}_{N}(t)\}\\ 0&,\>\>\>{\mathrm{o.w.}}\end{array}\right.$$

$$Q_i^k(t)$$

$$C_i^k(t)$$

$$n_i^k(T)=\sum _{t=1}^T{\mathbb{I}}_{\{{\phi }_k^t=i\}}\le \sum _{t=1}^T{\mathbb{I}}_{\{Q_i^k(t)\ge Q_{i^{\ast }}^k(t)\}}.$$

$$\displaystyle\mathbb{I}_{\{Q_{i}^{k}(t)>Q_{i^{*}}^{k}(t)\}}=\left\{\begin{array}[]{cl}1&,\>\>\>Q^{k}_{i}(t)\geq Q^{k}_{i^{*}}(t)\\ 0&,\>\>\>{\mathrm{o.w.}}\end{array}\right.$$

$$\mathbb{E}\left(n_i^k(T)\right)\le \sum _{t=1}^T\mathbb{P}\left(Q_i^k(t)\ge Q_{i^{\ast }}^k(t)\right).$$

$$\mathbb{E}\left(R_i^k(T)\right)={\Delta }_i\mathbb{E}\left(n_i^k(T)\right).$$

$$\left\{Q_i^k(t)\ge Q_{i^{\ast }}^k(t)\right\}\subseteq \left\{{\mu }_{i^{\ast }}<{\mu }_i+2C_i^k(t)\right\}$$

$$\cup \left\{{\hat{\mu }}_{i^{\ast }}^k\le {\mu }_{i^{\ast }}-C_{i^{\ast }}^k(t)\right\}\cup \left\{{\hat{\mu }}_i^k\ge {\mu }_i+C_i^k(t)\right\}$$

$$\mathbb{E}\left(n_i^k(T)\right)\le \sum _{t=1}^T\mathbb{P}\left({\mu }_{i^{\ast }}<{\mu }_i+2C_i^k(t)\right)$$

$$+\sum _{t=1}^T\mathbb{P}\left({\hat{\mu }}_{i^{\ast }}^k(t)\le {\mu }_{i^{\ast }}-C_{i^{\ast }}^k(t)\right)+\sum _{t=1}^T\mathbb{P}\left({\hat{\mu }}_i^k(t)\ge {\mu }_i+C_i^k(t)\right).$$

$$\mathbb{P}\left({\hat{\mu }}_i^k(t)-{\mu }_i>\sqrt{\frac{\vartheta }{N_i^k(t)}}\right)\le \frac{\nu \log Kt}{\exp (2\kappa \vartheta )}$$

$$\nu =\frac{1}{\log \zeta },\kappa =\frac{1}{{\sigma }_i^2{\left({\zeta }^{\frac{1}{4}}+{\zeta }^{-\frac{1}{4}}\right)}^2}.$$

$$\displaystyle\mathbb{P}\left(\Big{|}\hat{\mu}_{i}^{k}(t)-{\mu}_{i}\Big{|}>\sqrt{\frac{\vartheta}{N_{i}^{k}(t)}}\right)\leq\frac{\nu\log Kt}{\exp(2\kappa\vartheta)}.$$

$$\displaystyle\mathbb{P}\left(\Big{|}\hat{\mu}_{i}^{k}(t)-{\mu}_{i}\Big{|}>\sigma_{i}\sqrt{\frac{2\left(\delta(\xi)+\delta^{\prime}(\epsilon)\right)\log t}{N_{i}^{k}(t)}}\right)\leq\frac{\nu\log Kt}{t^{\delta(\xi)}}.$$

$$\frac{{\Delta }_i^2}{4{\sigma }_i^2}{\left(N_i^k(t)\right)}^2-2(\xi +1)\log t\left(N_i^k(t)+f(t)\right)$$

$${\eta }_i(t)=\frac{4{\sigma }_i^2(\xi +1)}{{\Delta }_i^2}\left(1+\sqrt{1+\frac{{\Delta }_i^2}{2{\sigma }_i^2(\xi +1)}\frac{f(t)}{\log t}}\right)\log t.$$

$$\sum _{t=1}^T\mathbb{P}\left(Q_i^k(t)\ge Q_{i^{\ast }}^k(t),N_i^k(t)<{\eta }_i(t)\right)\le {\eta }_i(T).$$

$$\mathbb{E}\left(n_i^k(T)\right)\le \Gamma (\zeta ,\xi ,K)+\frac{1}{T^{\xi -1}\log \zeta }\left(\frac{\log K}{T\xi }+\frac{1}{\xi -1}\right)$$

$$+\frac{4{\sigma }_i^2(\xi +1)}{{\Delta }_i^2}\left(1+\sqrt{1+\frac{{\Delta }_i^2}{2{\sigma }_i^2(\xi +1)}\frac{f(T)}{\log T}}\right)\log T$$

$${ϵ}_p^k$$

$$Q_i^k(t)$$

$$\mathbb{E}\left(n_i^k(T)\right)\le \Gamma (\zeta ,\xi ,K)+\frac{1}{T^{\xi -1}\log \zeta }\left(\frac{\log K}{T\xi }+\frac{1}{\xi -1}\right)$$

$$+\frac{4{\sigma }_i^2(\xi +1)}{{\Delta }_i^2}\left(1+\sqrt{1+\frac{{\Delta }_i^2}{2{\sigma }_i^2(\xi +1)}\frac{{ϵ}_p^k}{\log T}}\right)\log T.$$

$$\mathbb{E}\left(R^k(T)\right)\le \sum _{i=1}^N{\Delta }_i\Gamma (\zeta ,\xi ,K)$$

$$+\sum _{i=1}^N\left(\frac{{\Delta }_i}{T^{\xi -1}\log \zeta }\left(\frac{\log K}{T\xi }+\frac{1}{\xi -1}\right)\right)$$

$$+\sum _{i=1}^N\frac{4{\sigma }_i^2(\xi +1)}{{\Delta }_i}\left(1+\sqrt{1+\frac{{\Delta }_i^2}{2{\sigma }_i^2(\xi +1)}\frac{{ϵ}_p^k}{\log T}}\right)\log T.$$

$${ϵ}_p^k\le \frac{1}{p_k+1},\forall k.$$

$$\frac{1}{p_k+1}<\frac{1}{p_{k^{\prime }}+1};\sqrt{\frac{1}{d_k}\sum _{j=1,j\in {\mathcal{V}}_{d_k}}^K{p}_j}<\sqrt{\frac{1}{d_{k^{\prime }}}\sum _{j=1,j\in {\mathcal{V}}_{d_{k^{\prime }}}}^K{p}_j}$$

$$E\left(\exp (\lambda (X_i-{\mu }_i))\right)\le \exp \left(\frac{{\lambda }^2({\sigma }_i^{\prime }{)}^2}{2}\right).$$

$$Y_i^k(\tau )$$

$$E\left(\exp (\lambda Y_i^k(\tau ))|{\mathcal{F}}_{\tau -1}\right)\le \exp \left(\frac{{\lambda }^2{\sigma }_i^2}{2}{T}_i^k(\tau )\right).$$

Full text

I Introduction

Animal and robotic foraging problems that involve searching over spatially distributed patches with uncertain distribution of resource (reward) result in the explore-exploit dilemma. Each animal or robotic forager chooses which patch to sample, sequentially in time. The dilemma at every time in the sequence is whether to choose a well sampled patch that is expected to reap the highest reward (exploit) or to choose a poorly sampled patch that is expected to reduce uncertainty and possibly identify an option with an even higher reward (explore). Choices to explore can be costly or risky since they may not return much reward, but successful exploiting requires sufficient information that comes from exploration. The challenge is sorting out protocols for when and where to explore versus when and where to exploit.

In decision theory, multi-armed bandit (MAB) problems serve as models that capture the salient features of the explore-exploit trade-off [1, 2]. Thus, advances in addressing MAB problems directly benefit understanding and solving foraging and search problems. The MAB problem is analogous to a scenario in which an agent is repeatedly faced with several different options, each returning uncertain reward, and aims to make a sequence of choices to maximize the cumulative reward [3]. This is equivalent to minimizing the cumulative regret [4].

In their seminal work, Lai and Robbins [4] established a lower bound for the expected cumulative regret in the finite time horizon case. Specifically, they derived a logarithmic lower bound for the expected number of times a sub-optimal option needs to be sampled by an optimal sampling rule. They also established a confidence bound and a sampling rule to achieve logarithmic cumulative regret. These results were further simplified in [5] by introducing a confidence bound using a sample mean based method. Improving on these results, a family of Upper Confidence Bound (UCB) algorithms for achieving asymptotic and uniform logarithmic cumulative regret were proposed in [6]. These UCB algorithms are based on the notion that minimizing the expected cumulative regret is realized by choosing an appropriate uncertainty model, which results in an optimal trade-off between reward gain and information gain through uncertainty.

For the standard MAB problem the reward associated with an option is considered to be an iid stochastic process. Therefore, in the frequentist setting, the natural way of estimating the expectation of the reward is to consider the sample average [4, 5, 6]. The papers [7, 8] present how to incorporate prior knowledge about reward expectation in the estimation step by leveraging the theory of conditional expectation in the Bayesian setting.

The papers [9, 10] extend this work to the multi-agent setting where the explore-exploit problem is defined for a group of agents. The objective is to understand how the performance of individuals or the group can benefit from inter-agent observations. A centralized multi-agent setting is considered in [9] and a decentralized setting is considered in [11]. The papers [12, 10] use a running consensus algorithm in which agents observe the estimates of their neighbors. A multi-agent multi-armed bandit (MAMAB) problem, where agents observe instantaneous rewards and actions in a leader-follower setting, is considered in [13, 14].

In the above works, agents observe their neighbors, defined according to a static network graph. In this paper we consider a MAMAB problem where agents observe instantaneous rewards and actions of their neighbors through stochastic interactions. Observing rewards and actions rather than estimates is motivated by the foraging success of social groups where agents can observe neighbors even when their neighbors don’t necessarily want to share what they know [15]. Further, we assume that each agent $k$ only observes its neighbors with some probability $p_{k}$. This provides a framework for evaluating efficiency and robustness to changes in communication in terms of the $p_{k}$.

The setting is formulated by defining an underlying undirected network graph and imposing directed observation probabilities on each edge. The underlying graph models the inherent observation constraints present in the network. Imposed observation probabilities capture the heterogeneous social effort of agents in observing neighbors. We introduce the notion of sociability of each agent $k$ as the likelihood $p_{k}$ that the agent observes its neighbors. We derive analytical upper bounds for expected cumulative regret and propose a measure to rank agents according to their relative performance as a function of each agent’s sociability and the sociability of its neighbors. We show that our model predicts how high performance requires an agent to have both high sociability and neighbors with low sociability. This is an important result: it implies that making an investment to observe neighbors may be worthwhile only if those neighbors are sufficiently explorative.

In Section II we introduce the MAMAB problem with time-varying (stochastic) observation structure. We propose an efficient sampling rule for an agent to maximize cumulative expected reward. We analyze the performance of the proposed sampling rule in Section III. In Section III-A we analytically upper bound the expected cumulative regret and in Section III-B we propose a measure to predict the ranks of agents according to their relative performance. In Section IV we provide numerical simulation results and computationally validate the analytical results. We conclude in Section V and provide additional mathematical details in the Appendix.

II Multi-agent Multi-armed Bandit Problem

Consider a MAMAB with $K$ agents and $N$ arms, which represent $N$ options (patches). Let $X_{i}$ be a sub-Gaussian random variable with variance proxy $(\sigma_{i}^{\prime})^{2}$ that denotes the reward associated with option $i\in\mathcal{I}\triangleq\{1,2,\ldots,N\}.$ Define $\mu_{i}$ as the expected reward $\mathbb{E}(X_{i})$ of option $i$. Let $i^{*}$ be the optimal option such that $\mu_{i^{*}}=\max\{\mu_{1},\ldots,\mu_{N}\}$. Each agent $k\in\{1,\ldots,K\}$ chooses one option at each time step with the goal of maximizing its cumulative reward, which is equivalent to minimizing its cumulative regret. We assume that the $\mu_{i}$ are unknown and the $\sigma_{i}^{\prime}$ are known to the agents (e.g., if $\sigma_{i}^{\prime}$ is the magnitude of sensor noise).

Let $\mathcal{G}(\mathcal{V},\mathcal{E})$ be an undirected graph that encodes the observation structure of the system. If there exists an observation link $e_{kj}$ from agent $k$ to agent $j$, $j,k\in\mathcal{V}$, then $\{k,j\},\{j,k\}\in\mathcal{E}$, $e_{kj}=e_{jk}=1,$ and we say agents $k$ and $j$ are neighbors. For social or robotic groups searching over physical space, observation links may exist between pairs of agents when they are sufficiently close to be visible (or otherwise observable) to each other. Let $d_{k}$ be the number of neighbors and $\mathcal{V}_{d_{k}}$ the set of neighbors of agent $k$.

Let $\varphi_{k}^{t}\in\mathcal{I}$ and $X_{i}^{t}$ be random variables that denote the option chosen and reward received by agent $k$ at time $t$, respectively. Let $X_{i}^{t}$, $\forall t$, be i.i.d copies of $X_{i}.$ Define $\sigma_{i}=\sqrt{K}\sigma_{i}^{\prime}.$ Consider the probability space $(\Omega,\mathcal{U},\mathbb{P})$ and the increasing sequence of subalgebras $\mathcal{F}_{0}\subset\mathcal{F}_{1}\cdots\subset\mathcal{F}_{t}\cdots\subset\mathcal{F}_{T-1}\subset\mathcal{U}$ for $t=0,1,\cdots,T$, where $\mathbb{P}$ is the probability measure on the sigma algebra $\mathcal{U}$ of $\Omega$. Here $\mathcal{F}_{t}$ is the sigma algebra generated by information available at time $t$. Let $\mathbb{I}_{\{\varphi_{k}^{t}=i\}}$ be a $\mathcal{F}_{t-1}$ measurable indicator random variable that takes the value one if option $i$ is chosen by agent $k$ at time $t$ and is zero otherwise. Define $\mathbb{I}^{t}_{\{k,j\}}$ to be an $\mathcal{F}_{t-1}$ measurable indicator random variable that takes the value one if agent $k$ can observe agent $j$ at time $t$ and is zero otherwise.

Let $p_{k}$ be the probability that agent $k$ observes the instantaneous actions and rewards of its neighbors. Then we have $\mathbb{E}\left(\mathbb{I}^{t}_{\{k,j\}}\right)=p_{k},\forall j\neq k$ such that $\{k,j\}\in\mathcal{E}.$ We let $\mathbb{I}^{t}_{\{k,k\}}=1$ and $\mathbb{I}^{t}_{\{k,j\}}=0,\forall j\neq k$ such that $\{k,j\}\notin\mathcal{E}.$ An agent $k$ that has high probability $p_{k}$ is more likely to obtain observations from its neighbors. We introduce the notion of an agent’s sociability to refer to its value of $p_{k}$: we interpret agents with high $p_{k}$ values to be more sociable and agents with low $p_{k}$ values to be less sociable.

In order to maximize the cumulative reward in the long run, agents need to both identify the best options through exploring and sample the best options through exploiting. Since agents with high sociability values are more likely to obtain a greater number of observations, they can identify the best options with less exploring. However the usefulness of their observations is affected by the sociability values of their neighbors. Better performance can be obtained by an agent when it observes neighbors that do a lot of exploring (because they are less sociable), as compared to when it observes agents that do a lot of exploiting (because they are more sociable). This is, the agent will be able to exploit more, without compromising performance, when it has neighbors that are less sociable.

Let the number of times agent $k$ samples option $i$ in $T$ trials be given by the $\mathcal{F}_{T-1}$ measurable random variable $n_{i}^{k}(T)\triangleq\sum_{t=1}^{T}\mathbb{I}_{\{\varphi_{k}^{t}=i\}}$. And let the total number of times that agent $k$ observes rewards from option $i$ be the $\mathcal{F}_{T-1}$ measurable random variable $N_{i}^{k}(T)$, given as

[TABLE]

We define a sampling rule for agent $k$ as follows. Let the $\mathcal{F}_{t}$ measurable random variable ${\hat{\mu}}_{i}^{k}(T)$ be the estimate of option $i$ by agent $k$ at time $t$. We define

[TABLE]

where $S_{i}^{k}(T)=\sum_{t=1}^{T}\sum_{j=1}^{K}X_{i}^{t}\mathbb{I}_{\{\varphi_{j}^{t}=i\}}\mathbb{I}^{t}_{\{k,j\}}$ is the total reward observed by agent $k$ from option $i$ in $T$ trials.

Definition 1

The sampling rule $\{\varphi_{k}^{t}\}_{1}^{T}$ for agent $k$ at time $t\in\{1,\ldots,T\}$ is defined as

[TABLE]

with

[TABLE]

where $\xi>1$ and $f(t)$ a sublogarithmic nondecreasing nonnegative function.

III Performance Analysis

In this section we proceed to analyze the performance of the proposed sampling rule by analyzing the expected cumulative regret of the agents. Recall that the expected cumulative regret depends on the expected number of times the agents sample suboptimal options and the goal of each agent is to maximize its individual cumulative reward.

III-A Regret Analysis

Let $i$ be a suboptimal option. The total number of times agent $k$ samples from option $i$ can be upper bounded as follows:

[TABLE]

Here $\mathbb{I}_{\{Q_{i}^{k}(t)>Q_{i^{*}}^{k}(t)\}}$ is an indicator function that takes value one if the objective function of option $i$ is greater than the objective function of the optimal option $i^{*}$ and zero otherwise, i.e.,

[TABLE]

Thus we have

[TABLE]

Let $R_{i}^{k}(T)$ be the cumulative regret of agent $k$ from option $i$. Define $\Delta_{i}\triangleq\mu_{i^{*}}-\mu_{i}.$ Then we have, from [16],

[TABLE]

We proceed to analyze the expected number of samples from suboptimal options as follows.

First we note that $\forall i,k,t$ we have

[TABLE]

Next we proceed to analyze concentration probability bounds on the estimates of options.

Theorem 1

For any $\zeta>1$ and for $\sigma_{i}>0$ there exists a $\vartheta>0$ such that

[TABLE]

where

[TABLE]

Proof of Theorem 1: See Appendix.

Using symmetry we conclude that

[TABLE]

Lemma 1

For $\vartheta=2\sigma_{i}^{2}\left(\delta(\xi)+\delta^{\prime}(\epsilon)\right)\log t$, with $\epsilon>0,\delta(\xi)>0,\delta^{\prime}(\epsilon)=\delta(\xi)\epsilon/4$, there exists a $\zeta>1$ such that

[TABLE]

Proof of Lemma 1

*Define $\Gamma(\zeta)=\frac{1}{\left(\zeta^{\frac{1}{4}}+\zeta^{-\frac{1}{4}}\right)^{2}}$ where $\zeta>1.$ $\Gamma(\zeta)$ is a monotonically decreasing function and $\lim_{\zeta\to 1}\Gamma(\zeta)=\frac{1}{4}$. This implies that $\kappa<\frac{1}{4\sigma_{i}^{2}}$ since $\kappa=\Gamma(\zeta)/\sigma_{i}^{2}$. Choose $\zeta>1$ such that $\kappa=\frac{1}{(4+\epsilon)\sigma_{i}^{2}},$ where $\epsilon>0.$ Then $\exists\delta^{\prime}(\epsilon)>0$ such that $4\kappa(\delta(\xi)+\delta^{\prime}(\epsilon))\sigma_{i}^{2}=\delta(\xi).$ This proves that $\exists\zeta>1$ such that $2\kappa\vartheta=\delta(\xi)\log t$. The lemma follows from Theorem 1. *

We proceed to upper bound the summation of the probabilities of the events $\left\{\mu_{i^{*}}<\mu_{i}+2C_{i}^{k}(t)\right\}$ for $t\in\{1,2,\ldots,T\}$ as follows. Using the equation (5) we have that the inequality $\mu_{i^{*}}<\mu_{i}+2C_{i}^{k}(t)$ implies

[TABLE]

This inequality does not hold for $N_{i}^{k}(t)>\eta_{i}(t)$, where

[TABLE]

Thus we have

[TABLE]

Using probability bounds given in Lemma 1 and equation (9) we prove our main result on performance bounds.

Theorem 2

The total expected number of times agent $k$ samples suboptimal option $i$ until time $T$ is upper bounded as

[TABLE]

where $\Gamma(\zeta,\xi,K)=\frac{1}{\log\zeta}(1+\log K)+\frac{1}{2^{\xi}\log\zeta}\left(\frac{\log K}{\xi}+\frac{2}{\xi-1}\right)$ and $\zeta,\xi>1.$

Proof of Theorem 2: See Appendix.

Since $f(T)$ is a sublogarithmic function, the expected number of suboptimal samples are logarithmically bounded.

III-B Performance Measure

In this section we provide a measure to rank the agents according to their relative performance. We motivate with the following two cases for a set of four agents where there is an underlying all-to-all observation structure:

[TABLE]

.

In Case 1, neighbors of agent 1 are not at all sociable and in Case 2 they are maximally sociable. Therefore neighbors of agent 1 explore more in Case 1 than in Case 2. As a result, in Case 1, agent 1 tends to obtain observations from neighbors about lesser known options and this allows agent 1 to exploit more. In Case 2, agent 1 tends to obtain observations from neighbors about well sampled options, and this forces agent 1 to explore more. As a result agent 1 performs better in Case 1 as illustrated in Figure 1.

With this intuition we propose a measure as follows. First we restrict our attention to a class of problems where the underlying observation structure of the agents is a symmetric $d-$regular graph. This means that $d_{k}=d,\>\forall k$, i.e., every agent has the same number of neighbors. The all-to-all graph is a special case of the class of $d-$regular graphs with $d=K-1.$ We also assume $p_{k}\in\{.05,.10,.15,\ldots,1\},\forall k.$

We define performance measure $\epsilon_{p}^{k}\in(0,1]$ for agent $k$ as

[TABLE]

Our goal is to show that a lower $\epsilon_{p}^{k}$ implies a lower cumulative regret and therefore higher performance for agent $k$. The measure $\epsilon_{p}^{k}$ is inversely related to agent $k$’s sociability $p_{k}$ and directly related to the sociability $p_{j}$ of the neighbors $j$ of agent $k$. It then makes sense intuitively that lower $\epsilon_{p}^{k}$ implies higher performance for agent $k$, since the higher the $p_{k}$ the more agent $k$ observes and the lower the $p_{j}$ the more its neighbors explore and the more valuable is their information.

We next design a protocol for each agent $k$ that depends on $\epsilon_{p}^{k}$ and show in Corollary 1 that the bound on agent $k$’s cumulative regret is directly related to $\epsilon_{p}^{k}$, which suggests that the ordering of agents by $\epsilon_{p}^{k}$ predicts the ordering of agents by performance. We plan to prove in a future publication that the same ordering is predicted even when the protocol does not depend on $\epsilon_{p}^{k}$. Simulations in Section IV provide validation of these assertions.

Let $f(t)=\epsilon_{p}^{k}.$ Then the objective function becomes

[TABLE]

This assumes that each agent $k$ knows the $p_{j}$ of each of its neighbors $j$. From Theorem 2, the expected number of times agent $k$ samples the suboptimal option $i$ is bounded as

[TABLE]

This suggests that lower $\epsilon_{p}^{k}$ values correspond to lower cumulative expected regret and hence better performance. Using the bound on $\mathbb{E}\left(n_{i}^{k}(T)\right)$ and equation (6) we upper bound the cumulative expected regret of agent $k$ as follows.

Corollary 1

Let $R^{k}(T)$ be the regret of agent $k$ up to time $T$. Then we have

[TABLE]

Expected cumulative regret is then logarithmically bounded.

The performance measure $\epsilon_{p}^{k}$ can be bounded as

[TABLE]

Accounting for the sociability of neighbors, as we have done, provides a more accurate and tighter bound than only considering individual sociability values. However, for an all-to-all observation structure the rank order can be predicted using only individual sociability values as shown next.

Lemma 2

Let $\mathcal{G}$ be an all-to-all graph. Let $p_{k}$, $p_{k^{\prime}}$ be the sociability of agents $k$, $k^{\prime}$ such that $p_{k}>p_{k^{\prime}}.$ Then $\epsilon_{p}^{k}<\epsilon_{p}^{k^{\prime}}.$

Proof of Lemma 2

Since $p_{k}>p_{k^{\prime}}$ and $d_{k}=d_{k^{\prime}}=K-1$ we have

[TABLE]

By equation (10), this proves that $\epsilon_{p}^{k}<\epsilon_{p}^{k^{\prime}}.$

IV Numerical Simulations

We ran numerical simulations to evaluate the performance of sampling rule (3)–(III-B) with $f(t)=\epsilon_{p}^{k}$ for agent $k$ and with $f(t)=\log\log t$. The results in both cases verify the accuracy of agent ranks predicted by the performance measure $\epsilon_{p}^{k}$: lower $\epsilon_{p}^{k}$ corresponds to lower cumulative regret, hence higher performance. We show plots in the case $f(t)=\epsilon_{p}^{k}$.

We consider 6 agents playing 10-armed bandit problems with two distinct observation structures: A) an all-to-all graph and B) a cyclic graph. In all simulations we let the reward distributions be Gaussian with variance $(\sigma_{i}^{\prime})^{2}=25$, $i=1,\ldots,10$, mean values given by

[TABLE]

,

and sociability values given by

[TABLE]

.

We provide results for 500 time steps with 1000 Monte Carlo simulations. We set the sampling rule parameter $\xi=1.1$.

IV-A All-to-all observation

The underlying observation graph structure is all-to-all, equivalently a 5-regular graph. We calculate the performance measure $\epsilon_{p}^{k}$ for each agent $k$ using equation (10):

[TABLE]

.

The best predicted performer is agent 5 with lowest performance measure $\epsilon_{p}^{5}=0.374$ and highest sociability value $p_{5}=1.0$. Second and third best predicted performers are agent 6 and agent 2, respectively, with second and third highest sociability. Agents 1 and 4 are ranked next with equal sociability and $\epsilon_{p}^{1}=\epsilon_{p}^{4}.$ The worst predicted performer is agent 3, with lowest sociability and highest performance measure $\epsilon_{p}^{3}$. These predictions on performance ranking are verified in the simulation results of Figure 2. The results also verify that for an underlying all-to-all graph, the performance rank ordering is predicted by the sociability rank ordering.

IV-B Cyclic observation

The underlying observation graph structure is a cycle, equivalently a 2-regular graph, defined as $\{k,j\}\in\mathcal{E}$, where $|(k-j)\mod 6|=1$. We calculate the $\epsilon_{p}^{k}$ using (10):

[TABLE]

.

The best predicted performer is agent 2 with the lowest performance measure $\epsilon_{p}^{2}=0.284,$ but not the highest sociability. In fact, agents 5 and 6 have higher sociability than agent 2. However, while all three agents 2, 5, and 6 have one neighbor with sociability 0.5, the other neighbor of agents 5 and 6 has sociability 0.9 and 1, respectively, whereas the other neighbor of agent 2 has sociability 0.05. The very low sociability of one of agent 2’s neighbors improves agent 2’s performance significantly enough that it outperforms agents 5 and 6. This result illustrates the important role of sociability of neighbors in an agent’s performance. Further, while agents 1 and 4 are indistinguishable by their own sociability, agent 4 has a neighbor with sociability 0.05 whereas agent 1 has neighbors with relatively high sociability. Therefore as predicted by the performance measure, agent 4 outperforms agent 1. Figure 3 validates the predicted rankings.

V Conclusions

We studied a MAMAB problem where agents observe instantaneous actions and rewards of their neighbors according to a stochastic network graph in which agents are distinguished by their sociability, defined as the probability of observing their neighbors. We derived an upper bound for expected cumulative regret of agents. We proposed a measure to predict relative performance ranking of the agents as a function of sociability of agents and their neighbors. We verified that having less sociable neighbors improves the performance of agents. Accuracy of the measure has been verified analytically through expected cumulative regret bounds and computationally through numerical simulations.

acknowledgment

The first author wishes to thank Peter Landgren for helpful comments during the preparation of this paper.

Proof of Theorem 1

Since $X_{i}$ is a sub-Gaussian random variable with variance proxy $(\sigma_{i}^{\prime})^{2},$ we have

[TABLE]

Define a new random variable such that

[TABLE]

Note that $E(Y_{i}^{k}(\tau))=E(Y_{i}^{k}(\tau)|\mathcal{F}_{\tau-1})=0.$ Let $Z_{i}^{k}(t)=\sum_{\tau=1}^{t}Y_{i}^{k}(\tau).$ Let $T_{i}^{k}(\tau)=\sum_{j=1}^{K}\mathbb{I}_{\{\varphi_{j}^{\tau}=i\}}\mathbb{I}^{\tau}_{\{k,j\}}$ For any $\lambda>0$

[TABLE]

$\mathbb{I}_{\{\varphi_{j}^{\tau}=i\}}$* is an $\mathcal{F}_{\tau-1}$ measurable random variable, and so*

[TABLE]

*Further, using the properties of conditional expectations *

[TABLE]

Thus we see that

[TABLE]

The rest of the proof closely follows the papers [17, 18]. For clarity and completeness we include the main steps. Let $\zeta>1.$ Then $1\leq N_{i}^{k}(t)\leq\zeta^{D_{t}}$ where $D_{t}=\frac{\log Kt}{\log\zeta}.$ For $\lambda_{j}=\frac{2}{\sigma_{i}}\sqrt{\frac{\kappa\vartheta}{\zeta^{j-1/2}}}$ and $\zeta^{j-1}\leq N_{i}^{k}(t)\leq\zeta^{j}$ we have

[TABLE]

where $\kappa=\frac{1}{\sigma_{i}^{2}\left(\zeta^{\frac{1}{4}}+\zeta^{-\frac{1}{4}}\right)^{2}}.$

Recall from the Markov inequality that $\mathbb{P}(Y\geq a)\leq\frac{\mathbb{E}(Y)}{a}$ for any random variable $Y$. Thus,

[TABLE]

Proof of Theorem 2

From equations (8)–(9) we have

[TABLE]

Note that $\frac{N_{i}^{k}(t)+f(t)}{N_{i}^{k}(t)}\geq 1,\forall t\geq 1$. Using Lemma 1 and $\delta(\xi)=\xi+1-\delta^{\prime}(\epsilon)$ we have

[TABLE]

Since $\delta^{\prime}(\epsilon)$ can be made arbitrarily small, the summation term can be evaluated as follows:

[TABLE]

Thus we have

[TABLE]