Learning for Matching Game in Cooperative D2D Communication with Incomplete Information

TL;DR

This paper introduces a novel learning algorithm for cooperative D2D communication systems with incomplete information, enabling stable matching between CUs and D2D pairs despite limited channel data.

Contribution

It formulates the pairing as a matching game under incomplete information and proposes a new learning algorithm that guarantees convergence to a stable solution.

Findings

The proposed algorithm converges reliably under incomplete information.

It achieves stable matchings in cooperative D2D systems.

The approach outperforms traditional matching methods in realistic scenarios.

Abstract

This paper considers a cooperative device-to-device (D2D) communication system, where the D2D transmitters (DTs) act as relays to assist cellular users (CUs) in exchange for the opportunities to use licensed spectrum. Based on the interaction of each D2D pair and each CU, we formulate the pairing problem between multiple cues and multiple D2D pairs as a one-to-one matching game. Unlike most existing works, we consider a realistic scenario with incomplete channel information. Thus, each CU lacks enough information to establish its preference over D2D pairs. Therefore, traditional matching algorithms are not suitable for our scenario. To this end, we convert the matching game to an equivalent non-cooperative game, and then propose a novel learning algorithm, which converges to a stable matching.

Equations

$$R_m^C=\mathbb{E}\left[\ln \left(1+\frac{p_c{h}_{mb}}{n_0}\right)\right],$$

$$r_{mn}^C=\frac{1}{2}\left[\ln \left(1+\frac{p_c{h}_{mb}}{n_0}\right)+\ln \left(1+\frac{p_d{h}_{nb}^m}{n_0}\right)\right],$$

$${\tilde{R}}_{mn}^D({\alpha }_{mn})={\alpha }_{mn}\mathbb{E}\left[\ln \left(1+\frac{p_c{h}_{nn}^m}{n_0}\right)\right]\triangleq {\alpha }_{mn}{R}_{mn}^D,$$

$$U_{mn}^C({\alpha }_{mn})$$

$$U_{mn}^D({\alpha }_{mn})$$

$$\underset{{\alpha }_{mn}\in \mathcal{A}}{\max }$$

$$U_{mn}^C({\alpha }_{mn})>U_{min}^C,U_{mn}^D({\alpha }_{mn})>U_{min}^D,$$

$${\alpha }_{mn}^{\ast }(R_{mn}^C)={\left[\frac{R_{mn}^C-R_m^C}{2R_{mn}^C}\right]}_{{\alpha }_L}^{{\alpha }_U},$$

$${\pi }_m(b_m,{\mathbf{b}}_{-m})=\{\begin{array}{ll}{U}_{m{b}_m}^C({\alpha }_{m{b}_m}^{\ast })-\theta ,& \text{mbox}ifC{h}_{b_m}(\mathbf{b})=m,b_m\ne b_0,\\ -\theta ,& \text{mbox}ifC{h}_{b_m}(\mathbf{b})\ne m,b_m\ne b_0,\\ 0,& b_m=b_0,\end{array}$$

$${\hat{R}}_{mn}^C(t)={\hat{R}}_{mn}^C(t-1)+\lambda (t)\mathbf{I}(\widehat{C{h}_n}({\hat{\mathbf{b}}}^t)=m)(r_{mn}^C(t)-{\hat{R}}_{mn}^C(t-1)),$$

$${\hat{\alpha }}_{m^{\prime }n}^t=\{\begin{array}{ll}{\alpha }_{m^{\prime }}^t,& \text{mbox}if{b}_{m^{\prime }}^t=n\text{ and }{\alpha }_{m^{\prime }}^t\ne {\alpha }_e\\ {\hat{\alpha }}_{m^{\prime }n}^{t-1},& \text{mbox}otherwise.\end{array}$$

$${\hat{\pi }}_m(\mathbf{b})=\{\begin{array}{ll}(1-{\alpha }_{mn}^t){\hat{R}}_{mn}^C(t)-R_m^C-\theta ,& \text{mbox}ifC{h}_n(\hat{\mathbf{b}})=m\\ -\theta ,& \text{mbox}otherwise,\end{array}$$

$$\sum _{t=1}^{\infty }{P}_{mn}^t\ge \sum _{t=1}^{\infty }\zeta \epsilon (t)(1-\epsilon (t){)}^{M-1}\ge \sum _{t=1}^{\infty }\frac{\zeta {\epsilon }_0(1-{\epsilon }_0{)}^{M-1}}{t^{1/ML}}=\infty .$$

Full text

I Introduction

Recently, D2D communication has been extensively studied to provide better user experience. To implement this technology, one of the key issues is how to share licensed spectrum efficiently without degrading CUs’ performance greatly. We consider a cooperative D2D communication scheme, which exploits the advantages of cooperative relay and D2D communication [1]. The basic idea is that DTs act as relays for CUs in exchange for the transmission opportunities on the CUs’ channels. Thus, a win-win situation is achieved, which motivates CUs to share their spectrum with D2D pairs even if they have no surplus resource.

Most existing works[1, 2, 3, 4] assume complete information, such as channel state information (CSI). However, collecting global information incurs heavy overhead, and thus may be not practical in large-scale networks. Besides, some information may be difficult to acquire, such as the CSI between CUs and DTs. Moreover, the latency requirement of some applications is stringent, such as D2D-based vehicle-to-vehicle communications. These facts motivate us to study distributed resource allocation scheme with incomplete information, where agents make decisions independently based on local information.

Game theory provides a framework to study the interactions of autonomous agents. There have been many game theoretical solutions in D2D networks [5]. In our context, CUs have preferences over D2D pairs and vice versa. Matching theory offers a suitable tool to study the cooperation between competitive CUs and competitive D2D pairs. There have been some matching-based resource allocation schemes for D2D communication[6, 7, 8]. In this paper, we formulate the problem of pairing CUs with D2D pairs as a one-to-one matching game to seek a stable matching.

In the literature, authors of [9] have considered the incomplete information scenario, but do not investigate the pairing problem. Besides, similar cooperative scheme has been studied in cognitive radio networks recently[10, 11, 12, 13, 14, 15], where secondary users (SUs) relay primary users’ (PUs) traffic for rewards of the transmission opportunities. Some works adopt auction[10], dynamic Bayesian game[11], and Stackelberg game[12] to tackle the incomplete information. Moreover, the authors of [13] consider the incomplete information in the matching game model. However, above works [10, 11, 12, 13] assume PU has the knowledge of the relay rates, which depend on the SUs’ local information. In practice, such information is usually not known globally. In this paper, we consider a stronger incomplete information scenario, where CUs have no knowledge of the relay rates provided by the D2D pairs. The authors of [14, 15] consider the similar information assumption, but only consider single PU case. Instead, we consider the case with multiple CUs and multiple D2D pairs.

This paper focuses on the uplink resource sharing with incomplete information, because mobile devices are more likely to need help due to limited power budget. We formulate the pairing problem as a one-to-one matching game, based on the interaction between each CU and each D2D pair. Such interaction is described by Nash bargaining solution (NBS). Because the relay rates are unknown, CUs cannot establish preferences over D2D pairs. Thus, traditional matching algorithms, such as Gale-Shapley (GS) algorithm, are not suitable for our scenario. To the best of our knowledge, it is the first attempt to address the matching game with unknown preference. To overcome the difficulty, we convert the matching game to an equivalent non-cooperative game. At each period, each CU selects a D2D pair and a corresponding time allocation, and obtains a payoff as feedback. Based on the feedback, we propose a learning algorithm, which is proven to converge to a stable matching in probability. Moreover, the corresponding time allocation converges to the result of NBS with probability 1.

II System Model and Problem Formulation

II-A System Model

We consider uplink resource sharing of a single cell with a base station (BS) denoted by $b$ and $M$ CUs. The set of CUs is denoted by $\mathcal{M}$. Besides, there are $N$ D2D pairs, and the set of them is denoted by $\mathcal{N}$. Each D2D pair contains one DT and one D2D receiver (DR). In this paper, we assume $M\leq N$. However, the proposed algorithm can be applied to the case where $M>N$. CU $m$ has been assigned to one cellular channel, namely channel $m$. There is no dedicated channel for D2D pairs. Therefore, D2D pairs relay the uplink traffic in exchange for access to the cellular channels.

We assume that each CU is assisted by at most one D2D pair, and each D2D pair can relay at most one CU due to limited battery capacity[2]. Similar to [14, 15, 13], we take the decode-and-forward protocol with parallel channel coding [16] as an example. When CU $m$ cooperates with D2D pair $n$, the normalized frame consists of three phases, as shown in Fig.1. The first two phases both last $\frac{1-\alpha_{mn}}{2}$ and are used for the relay transmission for the CU. Specifically, CU $m$ broadcasts its data with power $p_{c}$ to the BS and DT $n$ at first. Then, DT $n$ forwards received signal to the BS with power $p_{d}$. The third phase lasts $\alpha_{mn}$ and is used by DT $n$ to transmit its data with power $p_{d}$ to DR $n$. We refer to $\alpha_{mn}\in\mathcal{A}\triangleq[\alpha_{L},\alpha_{U}]$ as time allocation.

The expected rate of CU $m$ in direct link is

[TABLE]

where $h_{mb}$ is the channel gain from CU $m$ to the BS and $n_{0}$ denotes the noise power.

For simplicity, we assume every DT can decode all the CUs’ data in the first phase. Thus, cooperating with D2D pair $n$, the rate of CU $m$ in the first two phases is

[TABLE]

where $h_{nb}^{m}$ is the channel gain from DT $n$ to the BS on channel $m$. Let $R^{C}_{mn}=\mathbb{E}[r^{C}_{mn}]$, and thus with time allocation $\alpha_{mn}$, the expected rate of CU $m$ during the entire frame is $\tilde{R}^{C}_{mn}=(1-\alpha_{mn})R^{C}_{mn}$. Moreover, the expected rate of D2D pair $n$ during the entire frame is given by

[TABLE]

where $h_{nn}^{m}$ is the channel gain of D2D pair $n$ on channel $m$. Assume that for each D2D link, the channel gains are i.i.d. across all the channels. Thus, we have ${R}^{D}_{mn}={R}^{D}_{m^{\prime}n},\forall m,\forall m^{\prime}\in\mathcal{M}$, and the value of ${R}^{D}_{mn}$ is denoted by $R^{D}_{n}$.

Information Assumption: CU $m$ only knows $R^{C}_{m}$ and has no knowledge of $R^{C}_{mn}$ and $R^{D}_{n}$, and D2D pair $n$ only knows $R^{D}_{n}$. After cooperating with D2D pair $n$ at period $t$, CU $m$ gets a sample $r^{C}_{mn}(t)$ following a fixed unknown distribution.

II-B Matching Based Framework

II-B1 Bargaining Game for CU-D2D Pair $(m,n)$

To incentivize CU and D2D pair to cooperate mutually, a bargaining game is used to characterize the interaction between them. If CU $m$ cooperates with D2D pair $n$, the CU’s utility $U^{C}_{mn}$ and the D2D pair’s utility $U^{D}_{mn}$ are defined as

[TABLE]

We use NBS as the bargaining outcome to determine the time allocation, and thus the cooperation satisfies some useful properties and is beneficial for both sides. Hence, based on the concept of NBS[17], the time allocation is given by the following problem

[TABLE]

where $U^{C}_{min}$ and $U^{D}_{min}$ are the CU’s and the D2D pair’s utilities respectively if they fail to reach an agreement. It is natural to set $U^{C}_{min}=U^{D}_{min}=0$. Thus, problem (6) is coincident with proportional fairness scheme. Constraint (6b) guarantees that both sides have incentive to participate in the cooperation. Solving problem (6), the optimal time allocation is given by

[TABLE]

where $[x]_{a}^{b}=\max(a,\min(x,b))$. Based on (7), the D2D pair with higher relay rate can obtain larger transmission time. Moreover, it is easy to verify that $U^{C}_{mn}(\alpha^{*}_{mn}(R^{C}_{mn}))$ is an increasing function of $R^{C}_{mn}$, which reflects the fact that the CU prefers to cooperate with the D2D pair offering higher relay rate. We will use $\alpha_{mn}^{*}(R^{C}_{mn})$ and $\alpha_{mn}^{*}$ interchangeably afterwards. When the problem (6) is infeasible, for convenience, we still let $\alpha_{mn}^{*}$ be the associated time allocation, and thus have $U^{C}_{mn}(\alpha_{mn}^{*})\leq 0$ in this case.

II-B2 Matching Game Model

CU and D2D pair can only be paired when they agree to cooperate mutually. Therefore, it is reasonable to model the pairing problem between the set of CUs and the set of D2D pairs as a one-to-one matching game under two-sided preferences. CU $m$ prefers D2D pair $n$ to D2D pair $n^{\prime}$ (i.e., $n\succ_{m}n^{\prime}$), if $U^{C}_{mn}(\alpha_{mn}^{*})>U^{C}_{mn^{\prime}}(\alpha_{mn^{\prime}}^{*})$. Similarly, D2D pair $n$ prefers CU $m$ to CU $m^{\prime}$ (i.e., $m\succ_{n}m^{\prime}$), if $U^{D}_{mn}(\alpha_{mn}^{*})>U^{D}_{m^{\prime}n}(\alpha_{m^{\prime}n}^{*})$, which is equivalent to $\alpha_{mn}^{*}>\alpha_{m^{\prime}n}^{*}$. Besides, if $U^{C}_{mn}(\alpha_{mn}^{*})>0$, D2D pair $n$ is acceptable to CU $m$, which is denoted by $n\succ_{m}\emptyset$.

Mathematically, a matching is a function $\mu:\mathcal{M}\cup\mathcal{N}\rightarrow\mathcal{M}\cup\mathcal{N}\cup\{\emptyset\}$, such that $\mu(m)=n$ if and only if $\mu(n)=m$, and $\mu(m)\in\mathcal{N}\cup\{\emptyset\}$, $\mu(n)\in\mathcal{M}\cup\{\emptyset\}$, for $\forall m\in\mathcal{M},\forall n\in\mathcal{N}$. Note that $\mu(x)=\emptyset$ implies that user $x$ is unmatched. We aim to seek a stable matching (SM), which is the major solution concept in matching game and defined as follows [18].

Definition 1

Let $\mu$ be a matching. A CU-D2D pair $(m,n)$ is a blocking pair if $\mu(m)\neq n$, $m\succ_{n}\mu(n)$ and $n\succ_{m}\mu(m)$. $\mu$ is individually rational if $\mu(m)\succ_{m}\emptyset,\forall m\in\mathcal{M}$. Thus, $\mu$ is stable if it is individually rational and there is no blocking pair.

SM captures the preferences of both sides and CUs will only be matched with acceptable D2D pairs in SM. The existence of SM is guaranteed[18]. The challenge is that each CU cannot establish its preference due to the unavailability of $R^{C}_{mn}$. Thus, the traditional GS algorithm [18] cannot be used to seek SMs.

III Learning for Matching with Incomplete Information

To overcome the difficulty, CU has to learn its preference from the interactions with D2D pairs. To this end, we convert the above matching game to an equivalent non-cooperative game, which enables us to exploit the rich learning techniques designed for non-cooperative game.

III-A Equivalent Non-cooperative Game Model

We convert the matching game to a non-cooperative game $\mathcal{G}=(\mathcal{M},\{\mathcal{B}_{m}\}_{m\in\mathcal{M}},\{Ch_{n}\}_{n\in\mathcal{N}},\{\pi_{m}\}_{m\in\mathcal{M}})$. Due to the priority of CUs on licensed spectrum, we let CUs be the players to propose to D2D pairs. The action of CU $m$ is to select a D2D pair $b_{m}\in\mathcal{N}$, which means CU $m$ proposes to cooperate with D2D pair $b_{m}$ with time allocation $\alpha^{*}_{mb_{m}}$. Each CU can refuse to cooperate with any D2D pairs, which is denoted by action $b_{0}$. Hence, the action set of CU $m$ is $\mathcal{B}_{m}=\mathcal{N}\cup\{b_{0}\}$. Given an action profile $\mathbf{b}=(b_{1},b_{2},\cdots,b_{M})$, each D2D pair selects the CU offering the maximal time allocation among the CUs proposing to it and rejects the others. If more than one CUs offers the maximal time allocation, the D2D pair will choose one of them based on a predefined rule. The CU chosen by D2D pair $n$ is denoted by $Ch_{n}(\mathbf{b})$111Mathematically, the choice function of D2D pair $n$ can be represented as $Ch_{n}(\mathbf{b})=\arg\max_{m\in\mathcal{M}_{n}(\mathbf{b})}\{\alpha^{*}_{mn}+w_{m}\}$, where $\mathcal{M}_{n}(\mathbf{b})$ is the set of CU proposing to D2D pair $n$ and $w_{m}$ is the bias assigned to CU $m$. The bias is determined by the predefined rule, and satisfies that if $\alpha^{*}_{mn}>\alpha^{*}_{m^{\prime}n}$, $\alpha^{*}_{mn}+w_{m}>\alpha^{*}_{m^{\prime}n}+w_{m^{\prime}}$ must hold. , which can reflect the preference of D2D pair $n$. Thus, the utility of CU $m$ is:

[TABLE]

where $\mathbf{b}_{-m}$ is the action profile of all the CUs except CU $m$, and $\theta>0$ is an arbitrarily small number and denotes the negotiation cost. Assume $\theta$ is sufficiently small so that $U^{C}_{mn}(\alpha^{*}_{mn})-\theta>0$ if $U^{C}_{mn}(\alpha^{*}_{mn})>0$. In the first case, $\theta$ makes sure that CUs only select acceptable D2D pairs at equilibriums. The first two cases imply acceptance and rejection of the CU’s proposal, respectively. The third case means that the CU refuses to cooperate with any D2D pairs.

Given an action profile $\mathbf{b}$, its associated matching $\mu_{\mathbf{b}}$ is obtained as follows: for $\forall m\in\mathcal{M},\forall n\in\mathcal{N}$, $\mu_{\mathbf{b}}(m)=n$ and $\mu_{\mathbf{b}}(n)=m$ if and only if $Ch_{n}(\mathbf{b})=m$. Hence, the relationship between the pure Nash equilibrium (PNE) of $\mathcal{G}$ and the SM can be stated as follows, which implies that an SM can be found via finding a PNE of $\mathcal{G}$.

Theorem 1

If action profile $\mathbf{b}$ is a PNE, $\mu_{\mathbf{b}}$ is an SM. Conversely, if $\mu$ is an SM, there is a PNE $\mathbf{b}$ such that $\mu_{\mathbf{b}}=\mu$.

Proof:

On the one hand, let $\mathbf{b}$ be a PNE. We will prove the stability of $\mu_{\mathbf{b}}$ by contradiction. The individual rationality is easy to verify. Suppose there is a blocking pair $(m,n)$ in $\mu_{\mathbf{b}}$. Thus, CU $m$ can take action $b^{\prime}_{m}=n$ to improve its utility, which violates our assumption. Therefore, $\mu_{\mathbf{b}}$ is stable.

On the other hand, let $\mu$ be an SM. We construct an action profile $\mathbf{b}$ as follows: for CU $m$, if $\mu(m)=n$, it takes action $b_{m}=n$ and action $b_{0}$ otherwise. We will prove that $\mathbf{b}$ is a PNE by contradiction. Suppose $\mathbf{b}$ is not a PNE, so there exists a CU $m$ deviating to take action $b^{\prime}_{m}\neq b_{m}$. If $b^{\prime}_{m}=b_{0}$, $\mu$ is not individually rational. Besides, If $b^{\prime}_{m}\in\mathcal{N}$, there is a blocking pair in $\mu$. Thus, $\mu$ is not stable, which violates our assumption. Therefore, $\mathbf{b}$ is a PNE. ∎

To develop the learning algorithm, we show that $\mathcal{G}$ is a weakly acyclic under better-replies game (WABRG), which enables us to adopt better-reply with inertia (BRI) learning algorithm [19] to find the PNE of $\mathcal{G}$. WABRG means that from any action profile, there is a better-reply path that terminates in a PNE in a finite number of steps. A better-reply path is a sequence of action profiles $(\mathbf{b}^{1},\mathbf{b}^{2},\cdots,\mathbf{b}^{t},\cdots)$, where for each $t$, there is a CU $m$ such that $b_{m}^{t}\neq b_{m}^{t-1}$, $\mathbf{b}_{-m}^{t}=\mathbf{b}_{-m}^{t-1}$ and $\pi_{m}(\mathbf{b}^{t})>\pi_{m}(\mathbf{b}^{t-1})$. In other words, in successive action profiles, only one CU changes its action to improve its utility.

Theorem 2

The proposed game $\mathcal{G}$ is a WABRG.

Proof:

Suppose $\mathbf{b}^{0}$ is not a PNE. We will construct a better-reply path that ends at a PNE to prove the theorem.

If there are any rejected CUs, we let them take action $b_{0}$ successively to obtain $\mathbf{b}^{1},\mathbf{b}^{2},\cdots,\mathbf{b}^{t_{1}}$, such that the CUs unmatched in $\mu_{\mathbf{b}^{t_{1}}}$ take action $b_{0}$. Furthermore, according to Theorem 2.33 in [18], there exists a finite sequence of matchings $\mu_{1},\mu_{2},\cdots,\mu_{k},\cdots,\mu_{K}$, where $\mu_{1}=\mu_{\mathbf{b}^{t_{1}}}$, $\mu_{K}$ is stable, and there is a blocking pair $(m_{k},n_{k})$ for $\mu_{k}$ such that $\mu_{k+1}$ is obtained from $\mu_{k}$ by satisfying the blocking pair $(m_{k},n_{k})$. Thus, we let CU $m_{1}$ select D2D pair $n_{1}$ to obtain $\mathbf{b}^{t_{1}+1}$. Similarly, we let rejected CUs take action $b_{0}$ successively to obtain $\mathbf{b}^{t_{1}+2},\mathbf{b}^{t_{1}+2},\cdots,\mathbf{b}^{t_{2}}$. Note that the above process will not change the associated matching, i.e., $\mu_{\mathbf{b}^{t_{2}}}=\mu_{2}$. Repeating the above process, we can obtain an action profile $\mathbf{b}^{t_{K}}$ such that $\mu_{\mathbf{b}^{t_{K}}}=\mu_{K}$ and the unmatched CUs in $\mu_{K}$ take action $b_{0}$. Note that $\mathbf{b}^{t_{K}}$ is exactly the constructed action profile in the proof of Theorem 1, so it must be a PNE. Besides, it is easy to find that the sequence $\mathbf{b}^{0},\mathbf{b}^{1},\cdots,\mathbf{b}^{t_{K}}$ is a better-reply path. Hence, we can verify Theorem 2. ∎

III-B Learning Algorithm

Because each CU’s utility is related to $R^{C}_{mn}$, each CU has to learn its utility from the interactions with D2D pairs. Furthermore, the action of CU $m$ can be redefined as a proposal $\hat{b}_{m}=(b_{m},\alpha^{*}_{mb_{m}})$, where the time allocation is unknown in the case of incomplete information. Therefore, CUs have to make proposals explicitly to help D2D pairs establish their preferences. Specifically, at each period $t$, based on history information, CU $m$ makes proposal $(b^{t}_{m},\alpha^{t}_{m})$, where $\alpha^{t}_{m}$ is calculated using the estimation of $R^{C}_{mb^{t}_{m}}$. Based on the proposal profile $\hat{\mathbf{b}}^{t}=(\hat{b}^{t}_{1},\hat{b}^{t}_{2},\cdots,\hat{b}^{t}_{M})$, D2D pair $n$ selects the CU offering the maximal time allocation, and the selected CU is denoted by $\hat{Ch}_{n}(\hat{\mathbf{b}}^{t})$. After cooperation with D2D pair $n$, CU $m$ can update its estimation using observation $r^{C}_{mn}$. Besides, to facilitate the learning process, CU $m$ can also choose the time allocation $\alpha_{e}=\alpha_{U}+\theta^{\prime}$ to make sure it has enough chances to cooperate with every D2D pair to obtain information, where $\theta^{\prime}>0$ is an arbitrary small number. Hence, with D2D pair $n$ selected, CU $m$ can choose $\alpha_{e}$ for exploration.

Combining BRI and Q-learning, we propose a novel learning algorithm. The entire algorithm is depicted in Algorithm 1 for some CU $m\in\mathcal{M}$. In step 3, CU $m$ randomly selects D2D pair for exploration with probability $\varepsilon(t)$, where step 3-a is used to announce time allocation $\alpha^{t}_{mn}$ to help other CUs estimate their utilities. In step 4, with probability $1-\varepsilon(t)$, CU $m$ adopts BRI to learn a PNE of $\mathcal{G}$ using estimated utility $\hat{\pi}_{m}$. In step 6, based on observations, CU $m$ updates its estimation of $R^{C}_{mn}$ in Q-learning way. Then, CU $m$ uses this updated estimation to calculate the associated time allocation in step 7. In step 9, CU $m$ uses other CUs’ announced time allocation and $\alpha^{t}_{mn}$ to estimate utility function.

Theorem 3

With $\varepsilon(t)=\varepsilon_{0}t^{-1/ML}$, the sequence $\{\alpha^{t}_{mn}\}$ converges to the true value $\alpha^{*}_{mn}$ with probability 1. Moreover, the algorithm converges to an SM in probability. Specifically, \lim_{t\rightarrow\infty}Pr\{\text{\mu_{\mathbf{b}^{t}} is an SM}\}=1, where $\mathbf{b}^{t}=(b^{t}_{1},b^{t}_{2},\cdots,b^{t}_{M})$.

Proof:

Let $P^{t}_{mn}$ denote the probability of CU $m$ cooperating with D2D pair $n$ at period $t$. Thus,

[TABLE]

So CU $m$ will cooperate with D2D pair $n$ infinitely often with probability 1. Based on [19], $\left\{\hat{R}^{C}_{mn}(t)\right\}$ converges to ${R}^{C}_{mn}$ with probability 1. Since $\alpha^{t}_{mn}$ is a continuous function of $\hat{R}^{C}_{mn}(t)$, we conclude that $\left\{\alpha^{t}_{mn}\right\}$ converges to $\alpha^{*}_{mn}$ with probability 1.

On the one hand, if we replace the estimated utility $\hat{\pi}_{m}$ with the true utility ${\pi}_{m}$ in EBRI-Q, the D2D pair selection process is exactly the stochastic BRI (SBRI) in [19]. Since $\mathcal{G}$ is a WABRG, using lemma 5.17 in [19], we have that $\lim_{t\rightarrow\infty}Pr\{\mathbf{b}^{t}\text{ is PNE}\}=1$ in this case. On the other hand, due to the step 3-a in EBRI-Q, the event that CU $m^{\prime}(m^{\prime}\neq m)$ announces its time allocation ${\alpha}^{t}_{m^{\prime}n}$ will happen infinitely often with probability 1. So $\hat{\alpha}^{t}_{m^{\prime}n}$ converges to ${\alpha}^{*}_{m^{\prime}n}$ with probability 1. Moreover, considering the convergence of $\hat{R}_{mn}(t)$ and $\alpha^{t}_{mn}$, the estimated utility will be sufficiently close to the true utility after an almost surely finite time. Thus, EBRI-Q will select D2D pairs with exactly the same probabilities as SBRI. Hence, based on Theorem 1, the convergence of $\mu_{\mathbf{b}^{t}}$ is verified. ∎

Remark 1

On the one hand, larger memory length $L$ improves the robustness to the exploration behavior of other D2D pairs, which may speed up the convergence rate of the algorithm. On the other hand, Theorem 3 implies that the exploration probability $\varepsilon(t)$ decays more slowly with larger $L$, which leads to slower convergence rate.

III-C Implementation Issues

At the beginning of each frame, CUs will send their proposals to the BS. Then, the BS broadcasts a proposal list containing all the CUs’ proposals at a dedicated channel. Meanwhile, all the D2D pairs will listen to this channel. After receiving CUs’ proposals, each D2D pair will accept one of them. Then, each matched D2D pair will send a feedback to the BS using the channel occupied by its matched CU. Based on these feedback, the BS obtains the final matching and informs the result to the CUs. Thus, each CU knows its partner and can begin its data transmission.

Except the above hand-shaking procedure, no extra overhead is needed in the proposed algorithm.Thus, each iteration has low signaling overhead. Note that (9)-(11) can be calculated in constant time. Moreover, the estimated utility is only needed in BRI. Thus, according to BRI, the algorithm only needs to estimate $\sum^{t-1}_{\tau=t-L}\pi_{m}(b,\mathbf{b}^{\tau}_{-m}),\forall b\in\mathcal{N}$. Therefore, the computational complexity of each iteration is $\mathcal{O}(LN)$.

IV Simulation Results

Simulation results are presented to evaluate the performance of the proposed algorithm. The channel gain is $\eta D^{-K}$, where $D$ is the distance between receiver and transmitter, $K=4$ is the path loss exponent and $\eta$ is fast fading with exponential distribution. The cell radius is 400 m. CUs are randomly distributed in an area of at least 300 m away from the BS. The distance between the DT and the BS is uniformly distributed between 150 and 250 m. The length of D2D link is uniformly distributed between 10 and 60 m. Besides, we set $n_{0}=-100$ dBm, $p_{c}=p_{d}=20$ mW, $\alpha_{L}=0.1$, $\alpha_{U}=0.5$, $\zeta=\varepsilon_{0}=0.1$ and $\xi=0.2$, and the length of memory in BRI is set to 4.

At first, we investigate the the convergence behavior of the proposed learning algorithm. For illustration purposes, we consider a small network with 2 CUs and 2 D2D pairs. There is only one SM, where CU 1 is matched with D2D pair 2 and CU 2 is matched with D2D pair 1. The results are given in Fig. 2 and Fig. 3. The results are averaged over 1000 simulations with the same topology. Fig. 2 presents the convergence of the time allocation estimation, where the estimation $\alpha^{t}_{mn}$ is normalized by the true value $\alpha^{*}_{mn}$. It is observed that the sequence $\left\{\alpha^{t}_{mn}\right\}$ converges to $\alpha^{*}_{mn}$ asymptotically, which is consistent with Theorem 3. The convergence of CUs’ behaviors is given in Fig. 3. It can be found that CU 1 and CU 2 could acquire their correct partners. This result implies that PNE or SM will be achieved eventually.

Next, we compare the proposed algorithm with other distributed algorithms in a larger network with 4 CUs and 5 D2D pairs. Fig. 4 shows the achieved system throughput over time for different algorithms. The results are averaged over 1000 simulations with different topologies. In the classical exploration-exploitation $\epsilon$-greedy algorithm, at each period, every CU selects the best D2D pairs so far with probability $1-\epsilon$, and some random D2D pair with probability $\epsilon$. Besides, the time allocation estimations are updated similarly to our algorithm. We take $\epsilon=0.1$ in the simulation. In the random algorithm, each CU selects D2D pair randomly and proposes $\alpha_{L}$ as time allocation to guarantee its performance. We present the non-cooperative scheme as well, where every CU takes action $b_{0}$. It can be observed that our algorithm yields significant gain over other learning algorithms. Besides, the performance loss due to incomplete information is small. It is also worth mentioning that the cooperative scheme achieves much better performance than non-cooperative scheme, which verifies the efficiency of the cooperative scheme.

V Conclusion

This paper considers a cooperative D2D communication system with incomplete information. We model the pairing problem between multiple CUs and multiple D2D pairs as a one-to-one matching game and propose a novel learning algorithm, which converges to a stable matching. The simulation results verify our analysis and show that the proposed algorithm outperforms the classical $\epsilon$-greedy algorithm. In the future work, the location information will be considered to divide CUs and D2D pairs into small groups to speed up the learning process. Moreover, the learning algorithm with faster convergence rate will also be investigated.