Optimal Distributed Channel Assignment in D2D Networks Using Learning in Noisy Potential Games

TL;DR

This paper introduces a novel distributed learning algorithm for optimal channel assignment in noisy D2D networks, effectively handling estimation noise and ensuring convergence to optimal solutions.

Contribution

It formulates CAP as a noisy potential game and proposes BLLA, the first distributed algorithm with proven convergence to optimal channel assignments under noise.

Findings

BLLA converges to optimal solutions in noisy environments.

Sum data rate increases with more channels and users.

BLLA outperforms better response algorithms in noisy settings.

Abstract

We present a novel solution for Channel Assignment Problem (CAP) in Device-to-Device (D2D) wireless networks that takes into account the throughput estimation noise. CAP is known to be NP-hard in the literature and there is no practical optimal learning algorithm that takes into account the estimation noise. In this paper, we first formulate the CAP as a stochastic optimization problem to maximize the expected sum data rate. To capture the estimation noise, CAP is modeled as a noisy potential game, a novel notion we introduce in this paper. Then, we propose a distributed Binary Log-linear Learning Algorithm (BLLA) that converges to the optimal channel assignments. Convergence of BLLA is proved for bounded and unbounded noise. Proofs for fixed and decreasing temperature parameter of BLLA are provided. A sufficient number of estimation samples is given that guarantees the convergence to the optimal state. We assess the performance of BLLA by extensive simulations, which show that the sum data rate increases with the number of channels and users. Contrary to the better response algorithm, the proposed algorithm achieves the optimal channel assignments distributively even in presence of estimation noise.

Tables

Table 1. TABLE I : Simulation parameters.

Parameter	Variable	Value
Number of orthogonal channels	$ℱ$	5
Channel bandwidth	$W_c$	180 KHz
Carrier frequency	$f_c$	2.6 GHz
Number of UEs	$\left|𝒟\right|$	20
Total transmit power of BS	$P_{\text{BS}}$	46 dBm
Transmit power of UE	$P_{\text{UE}}$	25 dBm
Minimum SINR	${\gamma }_{\min }$	-10 dB
Maximum SINR	${\gamma }_{\max }$	23 dB
Additive noise power per channel	$P_0$	$-174+10\log \left(W_i\right)$ dBm
Path-loss exponent	$\eta$	3.5
Shadowing variance	${\sigma }_{sh}$	6

Equations

$${\gamma }_i(c)=\frac{P_i{g}_i}{{\sum }_{j\in \mathcal{D}(c)\backslash i}{P}_j{g}_{j,i}+P_0},$$

$${\bar{c}}^{\ast }\in$$

$$U_i(a_i,a_{-i})-U_i(a_i^{\prime },a_{-i})=h(a_i,a_{-i})-h(a_i^{\prime },a_{-i}).$$

$${\hat{U}}_i^N(a_i,a_{-i})=\frac{1}{N}\sum _{k=1}^N{\hat{U}}_{i,k}(a_i,a_{-i}),$$

$${\hat{U}}_{i,k}(a_i,a_{-i})=\sum _{j\in \mathcal{D}(a_i)}{\hat{\nu }}_j(a_i,a_{-i})-\sum _{j\in \mathcal{D}(a_i)\backslash i}{\hat{\nu }}_j(a_i,a_{-i}),$$

$${\left(1+e^{{\Delta }_i^N/\tau }\right)}^{-1},$$

$$N\ge \left(\log \left(\frac{4}{\xi }\right)+\frac{2}{\tau }\right)\frac{{\ell }^2}{2{\left(1-\xi \right)}^2{\tau }^2},$$

$$N\ge \frac{\log \left(\frac{4}{\xi }\right)+\frac{2}{\tau }}{\log \left(\frac{e^{{\theta }^{\ast }\left(1-\xi \right)\tau }}{M({\theta }^{\ast })}\right)}.$$

$$0<\underset{\tau \to 0^{+}}{\lim }\frac{P_{\text{ab}}^{\tau }}{e^{-\frac{R_{\text{ab}}}{\tau }}}<\infty .$$

$$\underset{\tau \to 0}{\lim }\frac{f(\tau )}{g(\tau )e^{-\frac{\text{R}}{\tau }}}=1.$$

$$f(\tau )=g(\tau )e^{-\frac{\text{Res}(f)}{\tau }}+h(\tau ),$$

$$g_2(\tau )e^{-R_2/\tau }\in o\left(g_1(\tau )e^{-R_1/\tau }\right).$$

$$\underset{\tau \to 0}{\lim }\frac{g_2(\tau )e^{-R_2/\tau }}{g_1(\tau )e^{-R_1/\tau }}=\underset{\tau \to 0}{\lim }\frac{g_2(\tau )}{e^{-\left(R_2-k\right)/\tau }}{\left[\frac{g_1(\tau )}{e^{-\left(R_1-k\right)/\tau }}\right]}^{-1}.$$

$$f(\tau )=g_1(\tau )e^{-\frac{R_1}{\tau }}+h_1(\tau )=g_2(\tau )e^{-\frac{R_2}{\tau }}+h_2(\tau ),$$

$$1+\frac{h_1(\tau )}{g_1(\tau )e^{-\frac{R_1}{\tau }}}=\frac{g_2(\tau )e^{-\frac{R_2}{\tau }}}{g_1(\tau )e^{-\frac{R_1}{\tau }}}+\frac{h_2(\tau )}{g_1(\tau )e^{-\frac{R_1}{\tau }}}.$$

$$\underset{\tau \to 0}{\lim }{e}^{-\frac{\text{Res}(f)}{\tau }}=1.$$

$$f_1(\tau )$$

$$f_2(\tau )$$

$$\begin{gathered} f_1(\tau )+f_2(\tau )=g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}(1+\frac{h_1(\tau )}{g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}} \\ +\frac{g_2(\tau )e^{-\frac{\text{Res}(f_2)}{\tau }}}{g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}}+\frac{h_2(\tau )}{g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}}), \end{gathered}$$

$$f_1(\tau )+f_2(\tau )=g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}+h_3(\tau ),$$

$$f_1(\tau )+f_2(\tau )=e^{-\frac{\text{Res}(f_1)}{\tau }}\left[g_1(\tau )+g_2(\tau )\right]+h_1(\tau )+h_2(\tau ).$$

$$\text{Res}(f_1-f_2)=\text{Res}(f_1).$$

$$\begin{gathered} \underset{\tau \to 0}{\lim }\frac{f_1(\tau )}{g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}}\underset{\tau \to 0}{\lim }\frac{f_2(\tau )}{g_2(\tau )e^{-\frac{\text{Res}(f_2)}{\tau }}} \\ =\underset{\tau \to 0}{\lim }\frac{f_1(\tau )f_2(\tau )}{g_1(\tau )g_2(\tau )e^{-\frac{\text{Res}(f_1)+\text{Res}(f_1)}{\tau }}}=1. \end{gathered}$$

$$\underset{\tau \to 0}{\lim }\frac{f(\tau )}{g(\tau )e^{-\frac{\text{Res}(f)}{\tau }}}=1.$$

$$\underset{\tau \to 0}{\lim }\frac{\frac{1}{f(\tau )}}{\frac{1}{g(\tau )}{e}^{-\frac{-\text{Res}(f)}{\tau }}}=1.$$

$$f_1$$

$$g_1(\tau )e^{-\text{Res}(f_1)/\tau }+h_1(\tau )$$

$$1+\frac{h_1(\tau )}{g_1(\tau )e^{-\text{Res}(f_1)/\tau }}$$

$$1$$

$$=\underset{\tau \to 0}{\lim }\frac{f(\tau )}{g_{01}(\tau )g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}}\underset{\tau \to 0}{\lim }\frac{g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}}{f_1(\tau )},$$

$$=\underset{\tau \to 0}{\lim }\frac{f(\tau )}{g_{01}(\tau )g_1(\tau )e^{-\frac{\text{Res}(f_1)}{\tau }}},$$

$$P_{ab}^{\tau }=m_i(t)\frac{e^{\frac{1}{\tau }{U}_i(b)}}{e^{\frac{1}{\tau }{U}_i(a)}+e^{\frac{1}{\tau }{U}_i(b)}}.$$

$$=\text{Res}\left(m_i(t)\right)+\text{Res}\left(\frac{e^{\frac{1}{\tau }{U}_i(b)}}{e^{\frac{1}{\tau }{U}_i(a)}+e^{\frac{1}{\tau }{U}_i(b)}}\right),$$

Full text

I Introduction

Ever increasing demand for higher data rates of mobile users and scarcity of wireless frequency spectrum is making efficient utilization spectrum resources increasingly critical. Device-to-Device (D2D) networks increase the utilization of the spectrum resources by providing spatial spectrum reuse [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. In a D2D network, D2D users reuse the radio resources allocated to traditional cellular Users (UEs). The cellular UEs communicate with the Base Station (BS) while the D2D UEs communicate among themselves without or with limited help from BS. This is possible provided that the interference caused by the D2D UEs to cellular UEs is limited. A crucial problem in underlay D2D networks is thus to assign channels to UEs to increase its utilization while maintaining low interference.

Channel Assignment Problem (CAP) in D2D networks is challenging due to the lack of Channel State Information (CSI) of D2D links at the BS and because feedback overheads should be kept at a reasonable level. CSI estimation errors that are due to several factors such as randomly varying channel gain, feedback errors, feedback delay errors, and quantization errors [13] affect the performance of D2D network. It is, therefore, essential to have low feedback distributed solutions achieving optimal channel assignment while taking into account CSI or throughput estimation errors.

I-A Contributions

• •

Novel Approach: Our approach is to learn the optimal channel assignments in a D2D wireless network using a noisy potential game that takes into account the estimation noise. Distributed learning in a noisy environment for CAP is novel. We consider a Stochastic Optimization Problem (SOP) with the objective to maximize the expected sum data rate of an underlay D2D network. We translate this problem into a noisy potential game. The notion of the noisy potential game is introduced to account for the fact that only noisy estimates of the utility are available to the players.

• •

Learning algorithm: We propose a distributed Binary Log-linear Learning Algorithm (BLLA) for a SOP. BLLA solves CAP to achieve an optimal channel assignment, which corresponds to the optimal Nash equilibrium of the game. The convergence of BLLA is proved for fixed temperature and decreasing temperature parameter. We provide a sufficient number of estimation samples that guarantees the convergence for both the cases of bounded noise and unbounded noise. Note that for SOPs, BLLA is distributed and more practical when compared to Stochastic Approximation (SA) [14], Finite Difference SA [15], and Simultaneous Perturbation SA [16] algorithms, which are centralized and may not be desirable in large networks. Note that compared to BLLA the algorithm in [17] considers only the fixed temperature.

• •

Simulations results: Extensive simulations show that BLLA achieves the maximum sum data rate of the network. It shows that BLLA tracks well the increase of sum data rate with the increase of UEs and with the increase of the number of channels. We also show that contrary to better response algorithm, BLLA converges to the optimum even in presence of estimation noise.

I-B Related literature survey and comparison

In this subsection, we discuss and compare different approaches for CAP in the literature. CAP in wireless networks is a standard problem and it is known to be NP-hard [18]. Extensive surveys of CAP can be found for underlay D2D networks [1] and in various contexts in [19, 20].

The CAP solution approaches adopted by the state-of-the-art channel assignment algorithms are dynamic programming [10], graph-theoretical and heuristic solutions [11, 12], game theory [3, 4, 5, 6, 7, 8, 9], linear programming (LP), non-linear programming (NLP), and Markov Random Field [19]. Other approaches for CAP are neural networks [21], simulated annealing [22], tabu search, genetic algorithms [20].

In [10], the authors jointly optimize the mode selection and channel assignment in a cellular network with underlay D2D communications in order to maximize the weighted sum rate. A dynamic programming (DP) algorithm is proposed but it is exponentially complex. Therefore, a suboptimal greedy algorithm is proposed. In contrast to our approach, this solution relies on explicit closed form expressions of sum data rate for different channel fading scenarios. Our method can be applied to any fading scenario as it is based on users measured throughput. In [11], a suboptimal graph-theoretical heuristic solution for CAP in D2D networks is proposed. The weighted signal sum is maximized using maximum-weighted bipartite matching and interference sum is minimized using minimum-weighted partitioning. This approach is centralized since the BS uses the partial CSI of all the UEs. On the contrary, our approach is distributed, optimal, and maximizes the sum throughput instead of signal sum. In [12], a heuristic algorithm is proposed for joint mode selection, channel allocation and power allocation in a D2D wireless network. Channel estimation is assumed to be perfect.

Different game-theoretic models such as non-cooperative games, coalition formation games, and auction games are used to study the radio resource allocation issues in D2D networks [3, 9, 4, 5, 6, 7, 8]. In [9], a game-theoretical reverse iterative combinatorial auction is proposed as the allocation mechanism. However, in this auction, the BS needs to have the bid from all the D2D links that may create a huge feedback overhead. In [4], a pricing mechanism is proposed to maximize the network throughput under QoS constraint. However, the algorithm proposed is a heuristic algorithm whose performance is only evaluated through simulations. In contrast, BLLA’s convergence is proven theoretically and confirmed through simulations. In [5], the uplink resource allocation problem for multiple D2D and cellular users is modeled as a coalition game. Convergence to a Nash equilibrium is proved. However, the equilibrium may be sub-optimal and inefficient. In [6, 7] also, the coalition formation algorithms proposed to jointly solve mode selection and spectrum sharing may not converge to an optimal coalition structure. The contract-based game theoretic mechanism proposed in [8] is evaluated through simulations only.

This paper is organized as follows. The system model and problem formulation are described in Section II. A noisy potential game framework is developed in Section III. BLLA and its convergence results are given in Section IV. Simulation and conclusions are presented in Sections V and VI, respectively. Proofs are in Appendix of our arXiv paper [23].

II D2D Cellular Network Model

In this section, we describe the D2D cellular network model as shown in Fig. 1. This figure shows downlink (DL) and uplink (UL) models. We consider a single base station (BS) and two types of UEs: $(i)$ cellular UEs (UECs) that communicate with the BS and $(ii)$ D2D UEs (UEDs) that communicate with other UEDs. The set of UEs is denoted as $\mathcal{D}$. We consider a set of orthogonal frequency channel bands $\mathcal{F}$. The UECs are assigned different channels by the BS, whereas UEDs reuse these channels. A UE transmits on a single channel. The UEs that transmit on the same channel $c\in\mathcal{F}$ cause interference to each other, the amount of which depends on channel gains between transmitters and receivers.

II-A Channel Model

We consider a channel model that captures the effect of path-loss, shadowing, and small-scale fading. Let denote $\mathcal{D}(c)$ as the set of UEs on channel $c\in\mathcal{F}$. Let $P_{i}$ and $P_{0}$ denote the transmit power of UE $i$ and noise power, respectively. The signal-to-interference-plus-noise ratio (SINR) at the receiver of UE $i$ on channel $c$ is given as:

[TABLE]

where $g_{i}$ is the channel power gain between UE $i$ and its receiver, $g_{j,i}$ is the channel power gain between UEs $i$ and $j$. These $g_{i}$ and $g_{j,i}$ take into account the path-loss, shadowing, and small-scale fading. The theoretical data rate $\nu_{i}(c)$ of UE $i$ on the channel $c$ of bandwidth $W_{c}$ is given by the classical Shannon capacity formula, $\nu_{i}(c)=W_{c}\log_{2}\left({1+\gamma_{i}(c)}\right).$

Note that the channel power gains $g_{i}$, $g_{i,j}$ are subject to random variations. These variations arise due to randomly varying channel gain, feedback errors, feedback delay errors, and quantization errors [13]. Therefore, all the quantities defined are in fact random variables. We denote $\hat{\nu}$ and $\nu$ as the estimated data rate and the expected data rate, respectively.

II-B Problem Formulation

Our objective is to maximize the expected sum data rate of the network by assigning channels to UEs. Let $\bar{c}=\left({c_{i},c_{-i}}\right)$ denotes a channel assignment vector where UE $i$ is assigned the channel $c_{i}\in\mathcal{F}$ and UEs other than UE $i$ are assigned the channel vector $c_{-i}\in\mathcal{F}^{\left\lvert{\mathcal{D}}\right\lvert-1}$. The estimated data rate of a UE depends on vector $\bar{c}$ and is denoted as $\hat{\nu}_{j}\left({\bar{c}}\right)$. The objective function is $\hat{\phi}\left({\bar{c}}\right)=\sum_{j\in\mathcal{D}}\hat{\nu}_{j}\left({\bar{c}}\right).$ Formally, CAP is stated as:

[TABLE]

where $\phi\left({\bar{c}}\right)=\mathbb{E}[\hat{\phi}\left({\bar{c}}\right)]$ is the expected value over all the randomness. We seek to maximize the average sum data rate by using only estimates of data rates. Hence, the above problem is a SOP [24]. In the next sections, we develop a general solution framework for this kind of SOPs.

III Noisy Potential Game Framework

In real scenarios, UEs don’t experience the theoretical data rate and have access only to estimates of their average throughput that is corrupted by noise. In order to develop a distributed solution to the CAP, we model the channel assignment problem (2) as a stochastic game.

Definition 1

[CAP game] A CAP game is defined by the tuple $\mathcal{\hat{G}}\coloneqq\{\mathcal{D},\{X_{i}\}_{i\in\mathcal{D}},\{\hat{U}_{i}\}_{i\in\mathcal{D}}\}$, where $\mathcal{D}$ is a set of UEs that are players of the game, $\left\{{X_{i}}\right\}_{i\in\mathcal{D}}$ are action sets consisting of orthogonal channels, $\hat{U}_{i}:X\rightarrow\mathcal{R}$ are random utility functions with finite expectation, and $X\coloneqq X_{1}\times X_{2}\times\ldots X_{\left\lvert{\mathcal{D}}\right\lvert}$.

An action profile $a\coloneqq\left({a_{i},a_{-i}}\right)$ where $a_{i}\in X_{i}$ is the action of player $i$ and $a_{-i}\in X_{-i}$ is the action set of all the players except player $i$. Note that the action vector $a\in X$ is the same as the channel assignment vector $\bar{c}$ and $X=\mathcal{F}^{\left\lvert{\mathcal{D}}\right\lvert}$.

Potential games are attractive class of games, using which distributed solutions to optimization problems can be designed. If the objective function of the optimization problem is aligned with the potential function, then global maximizers of the objective are also the optimal Nash Equilibria (NE) of the game. The optimal NEs are the maximizers of the potential function. Moreover, for potential games with deterministic utilities, an NE always exists and there exist algorithms that are guaranteed to converge to NEs or to the global maximizers of the potential function [25]. Let us thus recall the definition of potential games.

Definition 2

[Potential game] A game $\mathcal{G}\coloneqq\{\mathcal{D},\{X_{i}\}_{i\in\mathcal{D}},\{U_{i}\}_{i\in\mathcal{D}}\}$ is a (deterministic) potential game if there is a potential function $h:X\rightarrow\mathcal{R}$ such that $\forall i\in\mathcal{D}$, $\forall a_{i},a_{i}^{\prime}\in X_{i}$ and $\forall a_{-i}\in X_{-i}$,

[TABLE]

This framework cannot be directly used for our CAP game because of the random utilities. We thus propose in this paper a new class of games, namely noisy potential games.

Definition 3

[Noisy potential game] Let the expected utility of player $i$ is denoted as $U_{i}=\mathbb{E}[\hat{U}_{i}]$. The game $\mathcal{\hat{G}}\coloneqq\{\mathcal{D},\{X_{i}\}_{i\in\mathcal{D}},\{\hat{U}_{i}\}_{i\in\mathcal{D}}\}$ is a noisy potential game if the game $\mathcal{G}\coloneqq\{\mathcal{D},\{X_{i}\}_{i\in\mathcal{D}},\{U_{i}\}_{i\in\mathcal{D}}\}$ is a potential game.

We now design the utility function of the CAP game so as to obtain a noisy potential game and align the potential function with the objective function of the CAP optimization problem. We consider the following utility function which represents the marginal contribution of player $i$ to the global utility averaged over $N$ samples:

[TABLE]

where $N$ is the number of estimation samples, and $\hat{U}_{i,k}$ is given by:

[TABLE]

where $\hat{\nu}_{j}$ is the measured data rate of player $j$ and $\mathcal{D}(a_{i})=\left\{{j\in\mathcal{D}:a_{j}=a_{i}}\right\}$ is the set of UEs using the same channel as $i$. Note that random utility $\hat{U}_{i,k}$ may have a large variance but the variance of the utility $\hat{U}^{N}_{i}$ can be reduced by increasing the number of samples $N$. We will see in the next section that the number of samples $N$ must be designed carefully so as to preserve the convergence properties of potential games.

We have the following result. The proof is straightforward.

Proposition 1

A CAP game $\mathcal{\hat{G}}^{N}\coloneqq\{\mathcal{D},\{X_{i}\}_{i\in\mathcal{D}},\{\hat{U}^{N}_{i}\}_{i\in\mathcal{D}}\}$ with utilities defined in (4), (5) is a noisy potential game with potential function $\phi\left({a}\right)$.

In the rest of the paper, we consider the CAP noisy potential game $\mathcal{\hat{G}}^{N}$.

IV Learning in Presence of Noise

In this section, we first describe the proposed binary log-linear algorithm (BLLA) for learning in the presence of noise. Then, we give the results on convergence of BLLA.

The details of BLLA are described in Algorithm 1 and shown in Fig. 2. Each time slot is divided into two phases of size $N$ samples each (Phase I and Phase II). At the beginning of each time slot $t$, BS randomly selects a player $i$ and a trial action $\hat{a}_{i}\in X_{i}$ with uniform probability. Also, BS informs all the players $j$ such that $a_{j}(t-1)\in\{a_{i}(t-1),\hat{a}_{i}\}$ to estimate their data rates and feedback this information to the BS at the end of the two phases. Player $i$ plays action $a_{i}(t-1)$ and $\hat{a}_{i}$ during Phase I and Phase II, respectively. At the end of Phase II, all players on $a_{i}(t-1)$ and $\hat{a}_{i}$ feedback to the BS their two estimates of their sampled mean data rates corresponding to Phases I and II. BS calculates the utility of player $i$ according to (4) and selects an action from the set $\left\{{a_{i}(t-1),\hat{a}_{i}}\right\}$ according to (6), where $\tau(t)$ is a temperature parameter that governs the convergence properties of BLLA. Then, BS informs player $i$ with the selected action. This feedback requires only one bit. BLLA is distributed in nature because only a few players have to feedback to the BS.

IV-A Convergence of BLLA

In this subsection, we present the results of convergence of BLLA for both the cases of bounded and unbounded noise. BLLA generates an irreducible Markov chain over the action space of the CAP game $\mathcal{\hat{G}}^{N}$. However, as the parameter $\tau$ goes to zero, the stationary distribution concentrates on a few states. The states whose limit probability is strictly positive as $\tau$ goes to zero are called stochastically stable. It is known that for exact potential games the stochastically stable states of BLLA are the maximizers of the potential function [26]. We extend this result to noisy potential games.

Theorem 2

The stochastically stable states of BLLA are the global maximizers of the potential function $\phi(a)$ if one of the following holds.

The estimation noise is bounded in an interval of size $\ell$ and the number of estimation samples used are

[TABLE]

where $0<\xi<1$. 2. 2.

The estimation noise is unbounded with finite mean and variance. Let $M(\theta)$ be moment generating function of noise. Assume that $M(\theta)$ is finite. Let $\theta^{*}=\operatorname*{arg\,max}_{\theta}\theta\left({1-\xi}\right)\tau-\log\left({M(\theta)}\right)$. The number of samples used are

[TABLE]

Proof:

See Appendix -A. ∎

A small $N$ is desired for practical implementations. We choose the lowest $N$ that satisfies Theorem 2. In Theorem 2, we have a convergence in probability for fixed parameter $\tau$. In Theorem 3, we consider the case of decreasing parameter $\tau$ for which we obtain an almost sure convergence to optimal state as in simulated annealing with cooling schedule [27].

Theorem 3

Consider BLLA with a decreasing parameter $\tau(t)=1/\log(1+t)$, and the number of samples $N(\tau)$ is given by Theorem 2. Then, BLLA converges with probability $1$ to the global maximizer of the potential function.

Proof:

See Appendix -B. ∎

V Simulations

V-A Simulation Parameters

In this Section, we present simulation results on the downlink considering standard wireless system parameters shown in Table I. We consider that a BS is located at the center of a region of radius $200$ m. Among 20 UEs there are 5 UECs. The UECs have dedicated channels and no two UECs are on the same channel. These UECs serve as passive players of the game because they do not change their channel. The receivers of UED transmitters are located around them uniformly random over a region of radius $20$ m. The UEDs learn their channel with the help of the BS and hence are the active players of the game.

The variations of Rayleigh fading over time are considered as the noise component for all the simulations. Channel variations result in random UEs data rates. The data rates are bounded because the SINR is bounded between $\gamma_{\min}$ and $\gamma_{\max}$. We assume only bounded noise in the simulations where the noise interval is set to $\ell=1$ due to normalized utilities. Besides, the additive white Gaussian noise with power $P_{0}$ is considered.

V-B Simulation Results

In Fig. 3, convergence to the maximum sum data rate of BLLA is shown with fixed temperature $\tau=0.1$ and decreasing temperature $\tau(t)=0.1/\log\left({1+t}\right)$111Note that $\tau(t)=0.1/\log\left({1+t}\right)$ works well even though it is smaller than that given by Theorem 3. The reason being that the height of the highest local maximum is smaller than the global maximum [27]. We consider that height to be $10\%$ of the global maximum, which is reasonable.. The number of samples are calculated according to (7) for $\tau=0.1$ and $\xi=10^{-5}$. BLLA reaches the maximum sum data rate with both fixed and decreasing temperatures. However, it has more variations for fixed temperature. For decreasing temperature, the probability of staying at the maximum is higher.

To study the effect of temperature we show the performance of BLLA for different temperatures in Fig. 4. As before, the samples are calculated corresponding to different temperatures. For higher temperature $\tau=0.5$, BLLA exhibits huge variations. The probability of being at a local maximum decreases with increasing temperature. As temperature decreases, the variations also decrease. Also, the probability of being at a local maximum increases with decreasing temperature. Therefore, the temperature should be chosen carefully to obtain the desired performance. BLLA with smaller $\tau=0.05$ gives the desired performance.

We study the effect of the number of samples on the performance of BLLA in Fig. 5. If players take decisions after every single sample, i.e., if the estimation errors are ignored, BLLA exhibits large variations due to noise. As the number of samples increases the performance of BLLA improves. If the number of samples is taken according to Theorem 2 then BLLA provides high and stable sum data rate.

We now study the effect of the number of channels in Fig. 6. This plot shows the average sum data rate obtained from BLLA at the end of 500 iterations averaged over 1000 realizations. We see that the sum data rate increases as the number of channels increases. This is intuitive because the optimal channel assignment has lower interference per channel. As evident from the figure, BLLA correctly tracks this phenomenon.

We also study the performance of BLLA by varying the number of UEs in Fig. 7 for 10 orthogonal channels and 10 UECs. As before, the sum data rate is obtained from BLLA at the end of 500 iterations averaged over 1000 realizations. As the number of UEs increases (up to approximately 60 in the figure), the sum data rate first linearly increases because of the increasing traffic. A linear growth is observed as long as interference is controlled. Sixty is much larger than the number of available channels, which means that BLLA manages to assign frequencies in such a way that UEs do not interfere too much. After 60 UEs, the increase is reduced because interference significantly affects the sum data rate. BLLA exactly tracks this behaviour as evident from Fig. 7.

We now compare in Fig. 8 the performance of BLLA and better response (BR) algorithm, which accepts the trial action only if its utility is better than the current action. Best response algorithm, which is same as better response with two actions, was applied to CAP with the objective of minimizing the total interference in Wireless Sensor Networks (WSN) in [28]. The parameter of BLLA is $\tau(t)=0.1/\log\left({1+t}\right)$. Each curve in the figure is obtained by averaging over 1000 realizations of the algorithms. BR performs the worst when noise is not taken into account, which corresponds to one estimation sample case. When the number of samples is increased to $200$, BR improves. However, BLLA is better than BR. For 2000 samples, BR performance is the same as BLLA. It shows that the number of samples for BR has to be tuned carefully to obtain the desired performance. On the contrary, BLLA performs better with the fixed number of samples without any tuning. Note that theoretically BR needs an infinite number of samples. There is no theoretical guarantee for its convergence for a finite number of samples. On the contrary, as we have proved, BLLA has a theoretical guarantee of convergence with a finite number of samples.

VI Conclusions

A novel optimal solution for CAP in D2D wireless networks that takes into account throughput estimation noise is presented. To capture this noise, a noisy potential game framework is introduced. A distributed Binary Log-linear Learning Algorithm that achieves the optimal channel assignments is proposed. A sufficient number of estimation samples that guarantees the convergence in both cases of bounded noise and unbounded noise are given that are validated using simulations. BLLA achieves the optimal sum data rate. The sum data rate increases with the number of channels and with the number of UEs. BLLA performs better than better response algorithm.

-A Proof of Convergence of BLLA with Fixed $\tau$

If the utilities of the CAP game are deterministic and without noise then the CAP game becomes an exact potential game. For an exact potential game, the stochastically stable states of BLLA are the maximisers of potential [26, 17, 29]. We prove the same even for a noisy potential CAP game $\mathcal{\hat{G}}^{N}$ if the number of samples is chosen carefully. Our proof approach is to show that for a particular number of samples the resistance BLLA with estimated utilities is same as that of with the deterministic utilities222In all the proofs, the considered utilities are normalized by the maximum potential $\phi_{\max}$.. This kind of proof idea based on resistance is similar to that of [30, 17].

We first recall the computation of the resistance of BLLA in a deterministic potential game, as in [26]. Let consider the CAP game $\mathcal{G}\coloneqq\left\{{\mathcal{D},\left\{{X_{i}}\right\}_{i\in\mathcal{D}},\left\{{U_{i}}\right\}_{i\in\mathcal{D}}}\right\}$, with expected utilities $U_{i}={\mathbb{E}}\!\left[{\hat{U}^{N}_{i}}\right]$. It is an exact potential game. BLLA induces a regular Markov process over the action space $X$ of $\mathcal{G}$ [26, 17, 29]. Let denote $P^{\tau}$ as the transition matrix of the regular Markov process.

Definition 4** (Resistance of transition [26])**

Let $a=\left({a_{i},a_{-i}}\right)$ and $b=\left({a^{\prime}_{i},a_{-i}}\right)$ be action profiles such that only player $i$ changes its action. Let $P_{\text{ab}}^{\tau}$ be a strictly positive probability transition function with the parameter $\tau$. A non negative number $R_{\text{ab}}$ is the resistance of transition $a\to b$ if

[TABLE]

To develop easy rules to compute the resistance of a function we give a generalised definition of resistance in the following. Let $``o"$ and $``\omega"$ denote little "o" order and little omega order, respectively. We call function $g(\tau)$ a sub-exponential function that satisfies $g\in o\left({e^{k/\tau}}\right)$ and $g\in\omega\left({e^{-k/\tau}}\right)$ for any $k>0$.

Definition 5** (Resistance of positive function)**

Let $f(\tau)$ be a strictly positive function. If there is a sub-exponential function $g(\tau)$ and a number $R$ such that (10) holds, then $R$ is unique (see Lemma 2) and is called the resistance of $f$, denoted by $Res(f)$.

[TABLE]

• Remark

Remark that Definition 5 includes Definition 4, in which $g(\tau)=\kappa,0<\kappa<\infty$.

• Remark

Note that (10) is equivalent to

[TABLE]

where $h(\tau)\in o\left({g(\tau)e^{-\frac{\text{Res}(f)}{\tau}}}\right)$.

Lemma 1

Consider any two sub-exponential functions $g_{1}(\tau)$ and $g_{2}(\tau)$. Consider two real numbers $R_{1}$ and $R_{2}$. If $R_{1}<R_{2}$ then

[TABLE]

Proof:

Let $k$ be a real number. Then

[TABLE]

The above limit goes to zero when we choose $R_{1}<k<R_{2}$. ∎

Lemma 2

If $\text{Res}(f)$ exists then it is unique.

Proof:

Assume that function $f$ have two different resistances $R_{1}$ and $R_{2}$. Then, from (11) we have

[TABLE]

where $h_{1}(\tau)\in o\left({g_{1}(\tau)e^{-\frac{R_{1}}{\tau}}}\right)$ and $h_{2}(\tau)\in o\left({g_{2}(\tau)e^{-\frac{R_{2}}{\tau}}}\right)$. Let $R_{1}<R_{2}$. Using Lemma 1, we have $h_{2}\in o\left({g_{1}(\tau)e^{-\frac{R_{1}}{\tau}}}\right)$. Rearranging the term in above equation, we have

[TABLE]

Using Lemma 1, we arrive at contradiction that $1=0$.

∎

The following Lemma gives useful rules for computing $\text{Res}(f)$.

Lemma 3

Let $f_{1}$ and $f_{2}$ be strictly positive real valued functions. Let $\kappa$ be a positive constant. If $\text{Res}(f_{1})$ and $\text{Res}(f_{2})$ exist then

$f(\tau)$* is sub-exponential if and only if $\text{Res}(f)=0$. In particular $\text{Res}(\kappa)=0$,* 2. 2.

$\text{Res}(e^{-\kappa/\tau})=\kappa$, 3. 3.

$\text{Res}(f_{1}+f_{2})=\min\left\{{\text{Res}(f_{1}),\text{Res}(f_{2})}\right\}$, 4. 4.

$\text{Res}(f_{1}-f_{2})=\text{Res}(f_{1}),\text{if }\text{Res}(f_{1})<\text{Res}(f_{2})$, 5. 5.

$\text{Res}(f_{1}f_{2})=\text{Res}(f_{1})+\text{Res}(f_{2})$, 6. 6.

$\text{Res}(\frac{1}{f})=-\text{Res}(f),\text{if Res}(f)\neq 0$. 7. 7.

If $f_{1}(\tau)\leq f_{2}(\tau)$, $\text{Res}(f_{1})$ and $\text{Res}(f_{2})$ exist then $\text{Res}(f_{2})\leq\text{Res}(f_{1})$. 8. 8.

Let $f_{1}(\tau)\leq f(\tau)\leq f_{2}(\tau)$. If $\text{Res}(f_{1})=\text{Res}(f_{2})$ then $\text{Res}(f)$ exists and $\text{Res}(f)=\text{Res}(f_{1})$.

• Remark

In Rule 4, if $\text{Res}(f_{1})=\text{Res}(f_{2})$ then we cannot compute $\text{Res}(f_{1}-f_{2})$ because the difference of sub-exponential functions may not be a sub-exponential function.

• Remark

In Rule 8, in general if $f_{1}(\tau)\leq f(\tau)\leq f_{2}(\tau)$ and $\text{Res}(f_{1})\neq\text{Res}(f_{2})$ then $\text{Res}(f)$ may not exists.

Proof:

Proof of rule 1: Let $f(\tau)$ be a sub-exponential function. Choosing $g(\tau)=f(\tau)$ from (10) we have

[TABLE]

Therefore, we have $\text{Res}(f)=0$.

Assume $\text{Res}(f)=0$. From (11), we have $f(\tau)=g(\tau)+h(\tau)$, which is a sub-exponential function.

Let $f(\tau)=\kappa$ and $g(\tau)=\kappa$ then $g(\tau)\in o\left({e^{\frac{\kappa}{\tau}}}\right)$ and $g(\tau)\in\omega\left({e^{-\frac{\kappa}{\tau}}}\right)$, $\kappa>0$. Substituting these in (10) we have $\text{Res}(\kappa)=0$.

Proof of rule 2: Substituting $f(\tau)=e^{-\kappa/\tau}$ and $g(\tau)=1$ in (10) we get $\text{Res}(e^{-\kappa/\tau})=\kappa$.

Proof of rule 3: Consider that $\text{Res}(f_{1})$ and $\text{Res}(f_{2})$ be the resistances of functions $f_{1}$ and $f_{2}$, respectively. We have

[TABLE]

where $h_{1}(\tau)\in o\left({g_{1}(\tau)e^{-\frac{\text{Res}(f_{1})}{\tau}}}\right)$, $h_{2}(\tau)\in o\left({g_{2}(\tau)e^{-\frac{\text{Res}(f_{2})}{\tau}}}\right)$.

The sum of two functions can be written as

[TABLE]

Consider the case when $\text{Res}(f_{1})<\text{Res}(f_{2})$. Using Lemma 1 we have $h_{2}\in o\left({g_{1}(\tau)e^{-\frac{\text{Res}(f_{1})}{\tau}}}\right)$. Therefore,

[TABLE]

where $h_{3}(\tau)\in o\left({g_{1}(\tau)e^{-\frac{\text{Res}(f_{1})}{\tau}}}\right)$. According to (11), we have $\text{Res}(f_{1}+f_{2})=\text{Res}(f_{1})=\min\left\{{\text{Res}(f_{1}),\text{Res}(f_{2})}\right\}$.

The case of $\text{Res}(f_{1})=\text{Res}(f_{2})$ leads to the same result as shown below.

[TABLE]

Note that sum of sub-exponential functions $g_{1}(\tau)+g_{2}(\tau)$ is a sub-exponential function. Observe that $h_{1}(\tau)+h_{2}(\tau)\in o\left({\left[{g_{1}(\tau)+g_{2}(\tau)}\right]e^{-\frac{\text{Res}(f_{1})}{\tau}}}\right)$. As in the previous case, according to (11) we have $\text{Res}(f_{1}+f_{2})=\text{Res}(f_{1})$

Proof of rule 4: Also, it can be shown similarly to the proof of rule 3 that if $\text{Res}(f_{1})<\text{Res}(f_{2})$ then

[TABLE]

Proof of rule 5:

[TABLE]

Therefore, $\text{Res}(f_{1}f_{2})=\text{Res}(f_{1})+\text{Res}(f_{2})$.

Proof of rule 6: Since $\text{Res}(f)$ exists we have

[TABLE]

Inversing both sides of the above equation, we get

[TABLE]

Therefore, we have $\text{Res}(\frac{1}{f})=-\text{Res}(f)$.

Proof of rule 7: Assume that $\text{Res}(f_{1})<\text{Res}(f_{2})$. Using Lemma 1, we have $g_{2}(\tau)e^{-\text{Res}(f_{2})/\tau}\in o\left({g_{1}(\tau)e^{-\text{Res}(f_{1})/\tau}}\right)$ and $h_{2}\in o\left({g_{1}(\tau)e^{-\text{Res}(f_{1})/\tau}}\right)$.

[TABLE]

As $\tau$ tends to zero we arrive at a contradiction that $1\leq 0$. Therefore, $\text{Res}(f_{1})\geq\text{Res}(f_{2})$.

Proof of rule 8: We have $1\leq\frac{f(\tau)}{f_{1}(\tau)}\leq\frac{f_{2}(\tau)}{f_{1}(\tau)}$ and $\text{Res}\left({\frac{f_{2}(\tau)}{f_{1}(\tau)}}\right)=\text{Res}(f_{1})-\text{Res}(f_{2})=0$. By Rule 1 $\frac{f_{2}(\tau)}{f_{1}(\tau)}$ is sub-exponential. This implies that $\frac{f(\tau)}{f_{1}(\tau)}$ is also sub-exponential Therefore, there exists $g_{01}(\tau)$ such that

[TABLE]

where the product $g_{01}(\tau)g_{1}(\tau)$ is also a sub-exponential function. Therefore, $\text{Res}(f)$ exists and $\text{Res}(f)=\text{Res}(f_{1})=\text{Res}(f_{2})$.

∎

As an illustration of application of the above rules we calculate the resistance of BLLA with deterministic utilities using the above rules. Let $m_{i}(t)$ denote the probability of choosing player $i$ to revise its action. In case of deterministic utilities, the transition probability $P^{\tau}_{ab}$ of BLLA is

[TABLE]

Let $\Delta_{i}=U_{i}(a)-U_{i}(b)$.

Using Lemma 3, we have $\text{Res}(P^{\tau}_{ab})$

[TABLE]

wherer $\Delta^{+}_{i}=\max\left\{{0,\Delta_{i}}\right\}$.

In the following, we show that the resistance of BLLA for the noisy potential CAP game $\mathcal{\hat{G}}^{N}$ with estimated utilities $\hat{U}^{N}_{i}$ is same as in (35). For this, we need the following lemma.

Lemma 4

Let denote $\Delta^{N}_{i}=\hat{U}^{N}_{i}(a)-\hat{U}^{N}_{i}(b),$ $\Delta_{i}=U_{i}(a)-U_{i}(b),$

[TABLE]

and consider the event $A^{\delta}=\left\{{\left\lvert{\Delta^{N}_{i}-\Delta_{i}}\right\lvert<\delta}\right\}$. Then

[TABLE]

Proof:

Notice that the probability of transition of BLLA from action $a$ to $b$ in noisy potential game $\mathcal{\hat{G}}^{N}$ is $p^{N}_{i}=\text{Pr}^{N}\left({a\to b}\right)$ given in (36) and in deterministic potential game is $p_{i}=\text{Pr}\left({a\to b}\right)$ given in (37). Using the law of total probability, we can write

[TABLE]

and

[TABLE]

It can be shown that the absolute value of the derivative of $p_{i}$ with respect to $\Delta_{i}$ is $\tau^{-1}p_{i}\left({1-p_{i}}\right)\leq\tau^{-1}p_{i}$. Therefore, we have

[TABLE]

Also, we bound \left\lvert{\text{ Pr}^{N}\left({a\to b\Big{\arrowvert}\bar{A}^{\delta}}\right)-p_{i}}\right\lvert\leq 2. Substituting, this and (41) in (40) we have (38). ∎

Proof:

Let denote the noise $Z_{i}=\hat{U}_{i}(a)-U_{i}(a)-\left({\hat{U}_{i}(b)-U_{i}(b)}\right)$. Using Hoefding inequality for bounded independent random variables, we have

[TABLE]

Substituting (42) in Lemma 4, we have

[TABLE]

Substituting the number of samples $N$ from (7) and $\delta=\left({1-\xi}\right)\tau$ in above, we have

[TABLE]

As before, the transition probability $P^{\tau}_{ab}$ of BLLA is

[TABLE]

In the following, we calculate the resistance of lower and upper bound of the above $P^{\tau}_{ab}$ using Lemma 3. Note that $\text{Res}(p_{i})=\Delta^{+}_{i}$, $\text{Res}(e^{-\frac{2}{\tau}})=2$, and $\Delta_{i}\leq 2$. The resistance of lower bound of $P^{\tau}_{ab}$ is $\text{Res}\left({\xi m_{i}(t)\left({p_{i}-e^{-\frac{2}{\tau}}}\right)}\right)$

[TABLE]

Similarly, the resistance of upper bound of $P^{\tau}_{ab}$ is $\text{Res}\left({\left({2-\xi}\right)m_{i}(t)p_{i}+\xi m_{i}(t)e^{-\frac{2}{\tau}}}\right)$

[TABLE]

Since both the bounds have the same resistance, by Rule 8 the resistance of $P^{\tau}_{ab}$ exists and is equal to $\text{Res}(p_{i})$. Therefore, the resistance of transitions of BLLA with bounded noise is same as in the case of without noise (35). ∎

Proof:

In this case, we use Chernoff bound to calculate $\text{Pr}\left({\bar{A}^{\delta}}\right)$ because of the unbounded noise as below. Let denote the noise $Z_{i}$ with moment generating function $M(\theta)$.

[TABLE]

where, (52) is obtained by assuming symmetric probability distribution of noise. However, for non-symmetric distribution a more complex expression can be obtained. Also, we used the Chernoff bound for independent and identically distributed random variables to obtain the equation (53).

Substituting (53), $\delta=\left({1-\xi}\right)\tau$, and the number of samples $N$ from (8) in Lemma 4, we have

[TABLE]

Following the same steps as before, we get that the resistance of transitions of BLLA with unbounded noise is same as in the case of without noise (35).

∎

-B Proof of Convergence of BLLA with Decreasing $\tau(t)$

We give the proof in the case of bounded noise. The proof for unbounded noise can be done similarly. The proof is divided into several lemmas. For a given parameter $\tau$, we fix $N(\tau)$ as in (7), and we consider $p(\tau)=p^{N(\tau)}$. Recall that $p(\tau)=\mathbb{E}\left[f(\Delta_{N},\tau)\right]$ with $f(\delta,\tau)=\left(1+\exp\left(\frac{\delta}{\tau}\right)\right)^{-1}$.

Lemma 5

Function $\displaystyle\frac{\partial f(\delta,\tau)}{\partial\tau}$ is odd, has the sign of $\delta$, is bounded in absolute value by $k/\tau$ for some $k>0$, and the maximum is attained (for positive value) at the point $a^{*}\tau$, where $a^{*}>0$.

Proof:

We have

[TABLE]

This is an odd function in $\delta$ that has the sign of $\delta$. Hence, we just consider the case $\delta>0$. Then

[TABLE]

with $Y=\exp(\delta/\tau)$. This is first positive and then negative when $\delta$ is positive. The maximum is reached when

[TABLE]

We claim that the maximum in $\delta$ is attained for $\delta^{*}=a^{*}\tau$, with $a^{*}>0$ a constant. Indeed, consider $\delta=a\tau$ with $a>0$ in (56), which gives

[TABLE]

Consider the function $g(a)=2+\exp(a)(1-a)+\exp(-a)(1+a)$. We have $g(0)=4$, and $g$ tends to $-\infty$ when $a$ goes to $\infty$. Furthermore, the derivative is $-a(\exp(a)+\exp(-a))$ which is strictly negative, hence there is a unique solution $a^{*}$ to the equation (56). Replacing $\delta$ by $\delta^{*}=a^{*}\tau$ in (55) yields:

[TABLE]

Hence the result follows with $\displaystyle k=\frac{a^{*}}{2+\exp(a^{*})+\exp(-a^{*})}$. ∎

Lemma 6

If $\delta>0$ (resp. $\delta<0$), then $p(\tau)$ is increasing (resp. decreasing) in the vicinity of $\tau=0$. Furthermore, $\left\lvert{p^{\prime}(\tau)}\right\lvert$ has resistance $\delta$.

Proof:

We consider $\delta>0$. The case $\delta<0$ is similar.

We will show that the derivative $p^{\prime}(\tau)$ is positive in the vicinity of 0. Previous lemma shows that $\displaystyle\frac{\partial f(\delta,\tau)}{\partial\tau}\leq k/\tau$. Since the constant function $k/\tau$ is integrable w.r.t. to the distribution of $\Delta_{N}$, then

[TABLE]

By previous lemma, the point reaching the maximum of $\displaystyle\frac{\partial f(\delta,\tau)}{\partial\tau}$ goes to zero when $\tau$ goes to zero, and the function is then decreasing. Hence, for any $\epsilon$, there is $\tau$ small enough such that the minimum (resp. maximum) of the derivative on the interval $[\delta-\epsilon,\delta+\epsilon]$ is attained at $\delta+\epsilon$ (resp. $\delta-\epsilon$). Consider the event

[TABLE]

Following the proof techniques used to show the convergence of BLLA

[TABLE]

In the above (60) is obtained by using Lemma 5 and (61) is obtained by choosing $\delta=(1-\xi)\tau$. Note that as in (55) the above second term is equivalent to $\displaystyle\frac{\delta+\epsilon}{\tau^{2}}\exp(-\frac{\delta+\epsilon}{\tau})$, which is a dominant term compared to $\displaystyle\frac{k\xi}{\tau}\exp(-\frac{2}{\tau})$ if $\tau$ is small enough. Hence the derivative is lower bounded by a positive function and then is positive. More, by choosing $\epsilon=\tau^{2}$, we see that the derivative is lower bounded by a function equivalent to $\frac{1}{\tau^{2}}e^{-\frac{\delta}{\tau}}$, which has the resistance $\delta$.

The upper bound is obtained with the following inequality:

[TABLE]

And by choosing $\epsilon=\tau^{2}$ we obtain the same equivalent function $\frac{1}{\tau^{2}}e^{-\frac{\delta}{\tau}}$, which has the resistance $\delta$,

Therefore, from Rule 8 we have that the resistance of $\left\lvert{p^{\prime}(\tau)}\right\lvert$ is $\delta$. ∎

Lemma 7

The non-homogeneous Markov chain generated by the BLLA algorithm with decreasing parameter $\tau(t)=\frac{1}{\log(1+t)}$ is weakly ergodic.

Proof:

The conditions of validity of Theorem 2 in [31] are checked by Lemma 6, Equation (54) and the classical choice of decreasing parameter $\tau$. More details about weak ergodicity can also be found in [32]. ∎

If a real valued function $f$ is defined on the interval $\left[{a,b}\right]$, $f$ is differentiable and its derivative $f^{\prime}$ is Riemann integrable then its total variation $V_{a}^{b}(f)$ is

[TABLE]

$f$ is bounded variation function if its total variation is finite i.e., $V_{a}^{b}(f)<\infty$. If the derivative $f^{\prime}$ is bounded then $V_{a}^{b}(f)<\infty$ and $f$ is bounded variation function.

Let $\pi(\tau)$ be the stationary distribution of the homogeneous Markov chain for a given $\tau$.

Lemma 8

$\pi(\tau)$* has a bounded derivative.*

Proof:

By the Markov chain tree theorem [33] for every state $c\in X$, we have $\pi_{c}(\tau)=\frac{u(c)}{\sum_{d\in X}u(d)}$ where

[TABLE]

$p_{e}(\tau)$ (36) is transition probability to state $e$ and $\mathcal{T}_{c}$ is the set of trees rooted in state $c$. Then

[TABLE]

Hence it suffices to show that $\displaystyle\frac{|u_{c}^{\prime}|}{\sum_{d}u_{d}}$ is bounded for all states $c$.

Let $\text{Res}\left({T}\right)$ denotes the total resistance of a tree $T$ and $R_{\min}$ denotes the resistance of the minimal resistance tree. By using Lemma 3, we obtain

[TABLE]

The derivative of transition probability $u^{\prime}_{c}(\tau)$ is obtained using (63) as

[TABLE]

where $p^{\prime}_{e}(\tau)$ is equivalent to $\exp(-|\delta|/\tau)/\tau^{2}$ by Lemma 6. The resistance of $u^{\prime}_{c}(\tau)$ is

[TABLE]

Since by Lemma 3 , $\text{Res}\left({p_{e}^{\prime}}\right)$ is $\left\lvert{\delta}\right\lvert$ that is same as that of $\text{Res}\left({p_{e}}\right)$ (35) if $\delta>0$ and is strictly greater if $\delta<0$. Therefore, we have $\text{Res}\left({p_{e}^{\prime}}\right)\geq\text{Res}\left({p_{e}}\right)$ and $\text{Res}\left({u^{\prime}_{c}}\right)\geq\text{Res}\left({\sum_{c}u_{c}(\tau)}\right)$. But the minimal resistance tree must contain a transition with null resistance (which corresponds to the best response).

Lemma 9

A minimum resistance tree must contain a transition with zero resistance.

Proof:

Assume that a minimum resistance tree $T_{\min}$ have all the transitions with non-zero resistance. Let the root of this tree be a state $s$ and let there be a transition from another state $s^{\prime}$ to $s$. Let $R_{s^{\prime}\to s}$ be a non-zero resistance of this transition. Note that the resistance of reverse transition $R_{s\to s^{\prime}}=0$ because it corresponds to the best response transition. Construct a new tree $T$ rooted at state $s^{\prime}$ by adding the transition $s\to s^{\prime}$ and removing the transition $s^{\prime}\to s$. The resistance of the tree $T$ is

[TABLE]

We arrive at a contradiction. Therefore, a minimum resistance tree must contain a transition with null resistance. ∎

Hence, the state $c$ at which $R_{\min}$ is reached contains at least a transition with $\delta\leq 0$. Therefore, $\text{Res}\left({u^{\prime}_{c}}\right)>R_{\min}$. Using Lemma 3, we have

[TABLE]

where $h_{1}(\tau)\in o\left({\exp\left({-\frac{\text{Res}\left({u^{\prime}_{c}}\right)-R_{\min}}{\tau}}\right)}\right)$. Observe from the above equation that $\frac{|u_{c}^{\prime}|}{\sum_{d}u_{d}}\to 0$ as $\tau$ goes to zero for all states $c$. This finally shows that the derivative $\left\lvert{\pi_{c}^{\prime}(\tau)}\right\lvert$ is bounded. ∎

Proof:

We check the assumptions of Theorem 1 in [34] are satisfied for the proof of Theorem 3. By Lemma 7, the algorithm generates a weakly ergodic non-homogeneous Markov chain. Lemma 8 shows that the stationary distribution $\pi(\tau)$ of the homogeneous Markov chain is a bounded variation function of $\tau$ (this is a direct consequence of derivative of $\pi(\tau)$ being bounded).

∎