Network Resource Allocation via Stochastic Subgradient Descent: Convergence Rate

TL;DR

This paper analyzes the convergence rate of stochastic dual subgradient descent in resource allocation problems, providing a precise characterization of the trade-off between convergence speed and accuracy in network settings.

Contribution

It introduces a novel stochastic bound on the objective gap and generalizes convergence results for stochastic subgradient methods in resource allocation.

Findings

Convergence rate of SDSD characterized with explicit trade-off.

New stochastic bound on the objective gap developed.

Simulation confirms theoretical rate results.

Abstract

This paper considers a general stochastic resource allocation problem that arises widely in wireless networks, cognitive radio, networks, smart-grid communications, and cross-layer design. The problem formulation involves expectations with respect to a collection of random variables with unknown distributions, representing exogenous quantities such as channel gain, user density, or spectrum occupancy. We consider the constant step-size stochastic dual subgradient descent (SDSD) method that has been widely used for online resource allocation in networks. The problem is solved in dual domain which results in a primal resource allocation subproblem at each time instant. The goal here is to characterize the non-asymptotic behavior of such stochastic resource allocations in an almost sure sense. It is well known that with a step size of $\epsilon$, {SDSD} converges to an $\mathcal{O}(\epsilon)$-sized neighborhood of the optimum. In practice however, there exists a trade-off between the rate of convergence and the choice of $\epsilon$. This paper establishes a convergence rate result for the SDSD algorithm that precisely characterizes this trade-off. {Towards this end, a novel stochastic bound on the gap between the objective function and the optimum is developed. The asymptotic behavior of the stochastic term is characterized in an almost sure sense, thereby generalizing the existing results for the {stochastic subgradient} methods.} For the stochastic resource allocation problem at hand, the result explicates the rate with which the allocated resources become near-optimal. As an application, the power and user-allocation problem in device-to-device networks is formulated and solved using the {SDSD} algorithm. Further intuition on the rate results is obtained from the verification of the regularity conditions and accompanying simulation results.

Tables

Table 1. TABLE I : Performance comparison ( 1000 1000 1000 times slots)

UE selection	Downloaded data (Mb)	Cost incurred	Avg. Utility minus penalty
Proposed	16316	4501	0.79
Random	12465	19951	-3.20
Opportunistic	3987	177	0.67

Equations

$$\underset{r,\{p^i\}}{\max }$$

$$r\le \mathbb{E}\left[\sum _{i\in {\mathcal{M}}_t}{R}_i(p_t^i,{\gamma }_t^i)\right]$$

$$\{p^i{\}}_{i\in \mathcal{M}}\in \mathcal{P},r_{\min }\le r\le r_{\max }$$

$$({\mathbf{P}}_1)({\mathbf{x}}^{\star },{\mathbf{p}}^{\star })$$

$$\mathbf{u}(\mathbf{x})+\mathbb{E}\left[\mathbf{v}(\mathbf{\phi },{\mathbf{p}}_{\mathbf{\phi }})\right]\ge 0,$$

$$\mathbf{x}\in \mathcal{X},\mathbf{p}\in \mathcal{P}.$$

$$\underset{r,\{p^i\}}{\max }$$

$$-z-\mathbb{E}\left[\sum _{i\in {\mathcal{M}}_t}{c}_t^i{p}_t^i\right]\ge 0$$

$$-r+\mathbb{E}\left[\sum _{i\in {\mathcal{M}}_t}{R}_i(p_t^i,{\gamma }_t^i)\right]\ge 0$$

$$\{p^i{\}}_{i\in \mathcal{M}}\in \mathcal{P},r_{\min }\le r\le r_{\max }$$

$$L(\mathbf{\lambda },\mathbf{x},\mathbf{p})=f_0(\mathbf{x})+⟨\mathbf{\lambda },\mathbb{E}\left[{\mathbf{s}}_t(\mathbf{x},{\mathbf{p}}_t)\right]⟩$$

$$g(\mathbf{\lambda })=\underset{\mathbf{x}\in \mathcal{X},\mathbf{p}\in \mathcal{P}}{\max }L(\mathbf{\lambda },\mathbf{x},\mathbf{p}).$$

$$\mathsf{D}=\underset{\mathbf{\lambda }\ge 0}{\min }$$

$$({\mathbf{x}}_t({\mathbf{\lambda }}_t),{\mathbf{p}}_t({\mathbf{\lambda }}_t))$$

$${\mathbf{\lambda }}_{t+1}$$

$$[{\mathcal{P}}_{\mathcal{L}}\left(\mathbf{\lambda }\right){]}_i=\{\begin{array}{ll}0& [\mathbf{\lambda }{]}_i<0\\ {\lambda }_{\max }& [\mathbf{\lambda }{]}_i>{\lambda }_{\max }\\ {\lambda }_i& 0\le [\mathbf{\lambda }{]}_i\le {\lambda }_{\max }\end{array}$$

$${\lambda }_{\max }\gg \frac{1}{\chi (\tilde{\mathbf{x}},\tilde{\mathbf{p}})}(g(\tilde{\mathbf{\lambda }})-f_0(\tilde{\mathbf{x}}))\ge \frac{(g(\tilde{\mathbf{\lambda }})-f_0(\tilde{\mathbf{x}}))}{G}$$

$$\underset{t\to \infty }{\lim }\frac{1}{t}\sum _{\tau =0}^{t-1}{\mathbf{s}}_{\tau }({\mathbf{x}}_{\tau }({\mathbf{\lambda }}_{\tau }),{\mathbf{p}}_{\tau }({\mathbf{\lambda }}_{\tau }))\ge 0$$

$$\underset{t\to \infty }{\lim }f({\bar{\mathbf{x}}}_t)\ge \mathsf{P}-(ϵ\overline{G^2}/2)$$

$${\mathbf{\lambda }}^{\star }$$

$${\mathbf{\lambda }}_{t+1}$$

$$\underset{0\le \tau \le T-1}{\min }g({\mathbf{\lambda }}_{\tau })\le \frac{1}{T}\sum _{\tau =0}^{T-1}g({\mathbf{\lambda }}_{\tau })$$

$${ϵ}^{\zeta -1/2}{C}_T(n,ϵ)\to 0\text{a.s. as }ϵ\to 0\text{ for fixed }n<\infty$$

$$n^{1/2-\zeta }{C}_T(n,ϵ)\to 0\text{a.s. as }n\to \infty \text{ for fixed }ϵ>0.$$

$$\mathbb{P}(C_t(n,ϵ)>{\nu }^{\zeta -1/2})<A\exp \left(-{\nu }^{2\zeta }\right)$$

$$\underset{0\le \tau \le T-1}{\min }g({\mathbf{\lambda }}_{\tau })\le \frac{1}{T}\sum _{\tau =0}^{T-1}g({\mathbf{\lambda }}_{\tau })$$

$$\Vert \frac{1}{T}\sum _{t\in \mathcal{T}}{\mathbf{e}}_t(\mathbf{\lambda })\Vert$$

$$\Vert \frac{1}{T}\sum _{t\in \mathcal{T}}{\mathbf{e}}_t(\mathbf{\lambda })-{\mathbf{e}}_t({\mathbf{\lambda }}^{\prime })\Vert$$

$$T^{1/2-\zeta }{L}_T^i(\mathcal{T})$$

$$\mathbb{P}(L_T^i(\mathcal{T})>T^{\zeta -1/2})$$

$$\Vert {\mathbf{e}}_t(\mathbf{\lambda })\Vert \le \Vert {\mathbf{f}}_t(\mathbf{\lambda })\Vert +\Vert \mathbf{f}(\mathbf{\lambda })\Vert \le 2G$$

$$e_t^j(\mathbf{\lambda })-e_t^j({\mathbf{\lambda }}^{\prime })=⟨\nabla e_t^j({\mathbf{\lambda }}_{{\beta }_j}),\mathbf{\lambda }-{\mathbf{\lambda }}^{\prime }⟩.$$

$$\frac{1}{T}\sum _{t\in \mathcal{T}}{\mathbf{e}}_t(\mathbf{\lambda })-{\mathbf{e}}_t({\mathbf{\lambda }}^{\prime })={\mathbf{E}}_T(\mathbf{\lambda },{\mathbf{\lambda }}^{\prime })(\mathbf{\lambda }-{\mathbf{\lambda }}^{\prime })$$

$$g({\mathbf{\lambda }}_t)$$

$${\delta }_{t+1}$$

$$\le {\Vert {\mathbf{\lambda }}_t-{\mathbf{\lambda }}^{\star }-ϵ{\mathbf{f}}_t({\mathbf{\lambda }}_t)\Vert }^2={\Vert {\mathbf{\lambda }}_t-{\mathbf{\lambda }}^{\star }\Vert }^2-2ϵ⟨{\mathbf{f}}_t({\mathbf{\lambda }}_t),{\mathbf{\lambda }}_t-{\mathbf{\lambda }}^{\star }⟩+{ϵ}^2{\Vert {\mathbf{f}}_t({\mathbf{\lambda }}_t)\Vert }^2$$

Full text

I Introduction

Resource allocation is a fundamental problem in economic theory that finds application in the design of wireless communication protocols [1], smart grid systems [2], and scheduling algorithms [3]. From an optimization perspective, the goal is to find the optimal allocation variables such as transmit power, bandwidth, operational schedule, or facility locations, so as to maximize the user satisfaction, minimize the cost, and satisfy all system constraints. The stochastic resource allocation problem arises in scenarios where the optimization problem includes random parameters with unknown distributions [4]. For such problems, the goal is to find an allocation policy that is feasible and optimal, on average [5] or with high probability [6]. Since the policy variable may be infinite dimensional , the problem is more tractable in the dual domain due to finite number of constraints, an aspect exploited by a number of algorithms; see e.g., [7, 8, 9] and references therein.

This paper focuses on the so-called online algorithms, where the allocation must occur every time the random parameter is realized and revealed. For each realization, the resource allocation adheres to the operational or “box” constraints, while the overall allocation policy is only asymptotically feasible and optimal. The dual problem may then be solved using the stochastic subgradient descent method, whose asymptotic behavior is well-known [10, 11]. Further justification for solving the problem in the dual domain was provided in [5, 7], where it was shown that such stochastic problems have zero duality gap under some mild conditions. The asymptotic feasibility and optimality of the allocated resources via primal averaging was also established in [8]. In a similar vein, the relationship between the stochastic and ’averaged’ dual iterates for the power and subcarrier allocation problem in OFDM was established in [9].

In resource allocation problems, it is possible for the environmental variables to change abruptly. This motivates the use of constant step sizes in stochastic algorithms, that obviate the need to restart the iterations whenever such a change occurs [12]. With a constant step size of ${\epsilon}$ however, it is well known that the stochastic iterates converge only to an $O({\epsilon})$-sized neighborhood of the optimum [8]. On the other hand, making ${\epsilon}$ arbitrarily small is also impractical, since it results in a slow convergence rate [13, 14]. The aforementioned trade-off between the rate of convergence and the asymptotic performance of the constant step-size stochastic dual subgradient descent (SDSD) algorithm is an important aspect that has not been studied explicitly.

The goal of this paper is to rigorously characterize the convergence rate of the SDSD algorithm in an almost sure sense. The key contribution of the paper is the development of stochastic bounds on the iterates produced by SDSD method, that explicate the role played by ${\epsilon}$ in ‘forgetting’ the initial conditions, and coming close to the optimum. To this end, the iterations are divided into epochs of duration $1/\epsilon$, and the optimality gap is analyzed for both fixed and arbitrarily small $\epsilon$. The main result of the paper is that the stochastic component of this gap goes to zero almost surely, either as the number of epochs go to infinity with fixed $\epsilon>0$, or as $\epsilon$ itself goes to zero. The bounds developed here specialize to the known asymptotic results, and are generally applicable to any problem solved via the SDSD method. To the best of our knowledge, these are also the first such convergence rate results for the constant step size SDSD algorithm. Corresponding results for the diminishing step size stochastic subgradient exist, but cannot be readily extended to the present case [15]. Likewise, the analysis in [16] can be extended to yield rate results that hold on average, but does not yield almost sure bounds. The analysis in the present work makes use of the strong law of large numbers directly, and is completely different from that in [15, 16].

As the second contribution, it is shown that the convergence rate results are readily applicable to the stochastic resource allocation problem of interest here. To further demonstrate the usefulness of the bounds, the paper details a contemporary application that uses mobile caching for improving service via device-to-device (D2D) communication [17, 18]. To this end, we consider the D2D edge caching framework where willing users offer data connectivity to highly mobile users experiencing spotty coverage. The problem is well-motivated for vehicular users who may download data from other users residing near the highway. The corresponding resource allocation problem is shown to satisfy the required regularity conditions, thereby demonstrating the flexibility afforded by the SDSD algorithm.

This paper is organized as follows. Sec. I-A lists some of the related work in this area, providing context to the current work. Sec. II starts with detailing the D2D edge caching problem and formulates the general network resource allocation problem. Sec. III discusses the various solution methodologies in the literature, including the stochastic subgradient descent (SSD) framework. Sec. IV provides the main results of the paper, stating the convergence rate results for both primal and dual problems. Sec. V further develops the D2D examples introduced in Sec. II, and verifies the different conditions required for the convergence results to hold. Simulation results on D2D example are provided in Sec. VI and Sec. VII concludes the paper.

The notation used here is as follows. Boldface letters denote column vectors, for which the inequalities and equalities are defined component-wise. The set of all real numbers is denoted by ${\mathbb{R}}$, and likewise the sets of non-negative reals, positive reals, and $K$-dimensional real vectors are denoted by ${\mathbb{R}}_{+}$, ${\mathbb{R}}_{++}$, and ${\mathbb{R}}^{K}$, respectively. Time indices are denoted by the subscripts $t$ and $\tau$. For a vector ${\mathbf{x}}$, $[{\mathbf{x}}]_{i}$ denotes its $i$-th entry, $\left\|{\mathbf{x}}\right\|$ denotes its $\ell_{2}$ norm, $\left\|{\mathbf{x}}\right\|_{p}$ denotes its $p$-th norm, for $p\in{\mathbb{R}}_{+}$, and ${\mathbf{x}}^{T}$ denotes the transposed row vector. The expectation operator is denoted by $\mathbb{E}\left[\cdot\right]$ and the inner product is denoted by $\langle\cdot,\cdot\rangle$. Finally, $[c]^{a}_{b}=\min(\max\{c,b\},a)$ and $[c]_{+}:=[c]_{0}^{\infty}$.

I-A Related work

Stochastic approximation algorithms have a long history, going back to the prototypical adaptive filtering algorithms by Robbins and Monro [19] and Widrow and Stearns [20], and have been studied extensively in the context of least mean square (LMS) and recursive least-squares (RLS) algorithms [21]. Stochastic gradient and subgradient methods have since been applied to neural networks [22], parameter tracking [23], large-scale machine learning [24, 15], and resource allocation problems [4]. Convergence of these algorithms is well known for various choices of the step-size parameter [25]. Convergence rate of the stochastic subgradient descent algorithm has been established for diminishing step size rules via non-asymptotic analysis [15]. However, not much is known about the convergence rate of the constant step-size counterpart, except for the fact that it exhibits linear convergence when far from the optimum, if the objective function is strongly convex [10]. The rate analysis presented here fills this gap for a class of convex problems that satisfy certain regularity conditions; see Sec. IV.

The use of dual subgradient algorithms for deterministic resource allocation was first popularized in [26]. Recovery of near-optimum allocations via primal averaging was proposed in [27], and the result was extended to stochastic resource allocation problems in [28, 8]. The corresponding convergence rate analysis for primal recovery was provided in [16], which also serves as a starting point for the analysis presented here. Note however that the extension of the rate results to stochastic problems is not trivial, and does not follow immediately from the result in [16]. The specific assumptions required to develop the bounds in this paper are inspired from those used in the context of stochastic approximation and averaging [29].

From a broader perspective, the work in this paper is also related to the backpressure algorithm, first proposed in the context of stochastic network optimization [30]. As shown in [31], the dual subgradient algorithm when applied to deterministic resource allocation problems, may also be viewed as the so-called drift-plus-penalty algorithm. The analysis in [31] however does not translate to convergence rate results for the SDSD algorithm.

The wireless caching framework utilizing D2D communications was first proposed in [17, 18], where the fundamental limits were analyzed. The system model described here builds upon the basic framework of [18] by formulating the problem within the resource allocation fabric, and adding some implementation details. The results presented here may also be applied to other stochastic resource allocation formulations, such as those in broadcast power allocation[4], OFDM [9], beamforming [32], cognitive radio networks [6], network utility maximization [33, 7, 8], demand-response in the smart grid [34], smart grid powerded green communications [35, 36] and energy harvesting [37].

II Problem Formulation

This section formulates the general stochastic network resource allocation problem. We begin with detailing a D2D caching example that is used to motivate the general problem.

II-A Motivation: D2D Mobile Caching

The D2D framework enables direct communication between nearby user equipments (UE), enabling greater spectrum utilization, higher energy efficiency, and increased overall throughput. The technology also allows unique solutions to connectivity problems that arise at the network edge or as a result of cellular congestion at overcrowded events. As an example, the D2D architecture proposed in [18] considers caching of popular content on mobile devices with excess storage. The content files are then available for download over a D2D link, allowing users to reach higher data rates, avoid congestion, and overcome coverage issues. By directly involving the smartphone equipped users into the process of content distribution, such an edge-caching solution not only cuts down the hardware provisioning costs but also promises better user experience.

This example builds upon the mobile caching framework proposed in [18]. Specifically, the mobile user equipment (MoUE) seeks to download a large file or stream media for a sufficiently long duration, while maintaining a reasonable download rate or quality of experience. Let $\mathcal{M}\!\!=\!\{\!1,\ldots,M\!\}$ be the set of mobile caches in the network and at time $t$, the requested chunk be available at $\mathcal{M}_{t}\!\!\subset\!\!\mathcal{M}$ unique mobile caches (devices) that are at close proximity to the user. The potential download rate $R_{i}(p_{t}^{i},\!\gamma_{t}^{i})$ depends on the power allocation $p_{t}^{i}$ at the $i$-th mobile cache, as well as on the channel gain $\gamma_{t}^{i}$ of the D2D links. The downloads also incur a cost $c_{t}^{i}\!\!\in\!\!{\mathbb{R}}_{++}$ per unit of transmit power $p_{t}^{i}$ for slot $t$. The costs could be in form of incentives provided to the mobile caches by the content delivery network (CDN) company, and/or directly charged to the MoUE in form of an “enhanced coverage” fee. At each time $t$, the MoUE selects a cache $i_{t}$ to download from, and obtains an average throughput of $r$ over time. Finally, the user satisfaction for the achieved average throughput $\!r$ is quantified through the concave utility function $U\!(r)$. Fig. 1 depicts an example scenario, where a MoUE connects to different UEs in order to download cached data, that would otherwise be available only from the base stations.

The resulting stochastic resource allocation problem is formulated as

[TABLE]

where the expectations are with respect to the random variables ${\boldsymbol{\varphi}}_{t}\!:=\!{(\mathcal{M}_{t},\{c^{i}_{t}\}_{i\in\mathcal{M}_{t}},\{\gamma^{i}_{t}\}_{i\in\mathcal{M}_{t}})}$. The optimization variables consist of the power allocation function $p^{i}$ and the rate variable $r$. The formulation of (1) follows the classical “utility minus penalty” maximization format common to network resource allocation problems [33]. The set of functions ${\mathcal{P}}$ is such that only one out of ${\{p_{t}^{i}\}_{i\in\mathcal{M}}}$ is non-zero for each $t$ (cf. Sec. V). Consequently, the summations in (1a) and (1b) consist of only one term for each $t$. The set ${\mathcal{P}}$ also specifies the maximum and minimum values of $p_{t}^{i}$ for each $t$. Finally, the constraint in (1b) ensures that the power allocated per-time slot is sufficient to satisfy the average rate requirement.

The specific form of the rate function depends on the wireless technology used by the users. For instance, under slow fading scenarios, the power allocation and user selection can occur every coherence interval. Since channel state information can be acquired easily, the users may employ adaptive modulation and coding in order to achieve a rate close to the ergodic capacity. Specifically, for the mobile device $i$, the potential transmission rate is of the form R_{i}(p_{t}^{i},\gamma_{t}^{i}):=W{\color[rgb]{0,0,0}\log_{2}}(1+p_{t}^{i}\gamma_{t}^{i}/\alpha) where $W$ is the bandwidth of the channel and $\alpha$ includes the effect of noise and interference as well as other impairments, such as the use of finite block length codes and imperfect channel state information at the transmitter.

More realistically, under fast fading scenarios, the power allocation must occur over intervals that are significantly longer than the coherence time. In this case, it is more reasonable to consider the average rate over several coherence intervals as \!R_{i}(p_{t}^{i},\!\gamma_{t}^{i})\!:=\!W\mathbb{E}_{h}\left[{\color[rgb]{0,0,0}\log_{2}}(1\!+\!p_{t}^{i}\gamma_{t}^{i}h_{i}/\alpha)\right] where $h_{i}$ is the small-scale fading gain [38]. In this case, the power allocation and user selection occur only on the basis of the average channel gain $\gamma_{t}^{i}$, which changes slowly. It is remarked that the system model considered here allows other forms of the rate function as well.

II-B Stochastic Resource Allocation

This section considers the more general stochastic resource allocation problem where the formulation involves expectations with respect to a collection of $q$ random variables with unknown distributions, denoted by ${\boldsymbol{\varphi}}\in{\mathbb{R}}^{q}$. Of particular interest are the problems arising in the context of wireless communications and networks, where ${\boldsymbol{\varphi}}$ captures the state of the system, and the formulation takes the form [7, 8, 6, 9]

[TABLE]

The optimization variables in ${(\mathbf{P}_{1})}$ comprise of the resource allocation variable ${\mathbf{x}}\in{\mathbb{R}}^{d}$ and the policy functions ${\mathbf{p}}:{\mathbb{R}}^{q}\rightarrow{\mathbb{R}}^{p}$. The objective function $f_{0}:{\mathbb{R}}^{d}\rightarrow{\mathbb{R}}$ is a concave function, while the set $\mathcal{X}$ is compact and convex. The vector-valued constraint function is defined as ${\mathbf{u}}({\mathbf{x}}):=[u_{1}({\mathbf{x}})\cdots u_{K}({\mathbf{x}})]^{T}$, where $\{u_{i}({\mathbf{x}}):{\mathbb{R}}^{d}\rightarrow{\mathbb{R}}\}_{i=1}^{K}$ are concave functions. In contrast, no such restriction is placed on the vector-valued function ${\mathbf{v}}:{\mathbb{R}}^{q}\times{\mathbb{R}}^{p}\rightarrow{\mathbb{R}}^{K}$ and the compact set of functions ${\mathcal{P}}$. The rate analysis in Sec. IV however does require the overall problem to satisfy certain regularity properties, such as Slater’s constraint qualification and differentiability of the subgradient error; see (A1)-(A4).

It can be seen that the D2D edge caching problem in (1) is a special case of (2)-(4). Introducing a scalar variable $z\in{\mathbb{R}}$, it is possible to write (1) equivalently as

[TABLE]

Comparing (5) with (2)-(4), we see that ${\mathbf{x}}=[r,z]^{T}$ and ${\boldsymbol{\varphi}}_{t}:={(\mathcal{M}_{t},\{c^{i}_{t}\}_{i\in\mathcal{M}_{t}},\{\gamma^{i}_{t}\}_{i\in\mathcal{M}_{t}})}$. Likewise the forms of vector functions ${\mathbf{u}}$ and ${\mathbf{v}}$ can be readily inferred.

Since the distribution of ${\boldsymbol{\varphi}}$ is also not known in advance, it is generally not possible to solve ${(\mathbf{P}_{1})}$ in an offline manner. The goal here is to solve ${(\mathbf{P}_{1})}$ in an online fashion by observing the realizations of the independent identically distributed (i.i.d.) process $\{{\boldsymbol{\varphi}}_{t}\}_{t\in{\mathbb{N}}_{0}}$, where ${\mathbb{N}}_{0}$ is the set of non-negative integers. For most problems of interest, such a framework also entails online allocation of resources for each time $t$. To this end, the class of algorithms considered here will output the sequence of vector pairs $\{{\mathbf{x}}_{t},\mathring{{\mathbf{p}}}_{t}\}$ for each $t$, for the purpose of allocating resources. For the sake of brevity, we will subsequently denote policy function ${\mathbf{p}}_{t}:={\mathbf{p}}_{{\boldsymbol{\varphi}}_{t}}$ and $\mathbf{s}_{t}({\mathbf{x}},{\mathbf{p}}_{t}):={\mathbf{u}}({\mathbf{x}})+{\mathbf{v}}({\boldsymbol{\varphi}}_{t},{\mathbf{p}}_{{\boldsymbol{\varphi}}_{t}})$, so that (3) can equivalently be written as $\mathbb{E}\left[\mathbf{s}_{t}({\mathbf{x}},{\mathbf{p}}_{t})\right]\geq 0$. Here, it is understood that the expectation is with respect to the random vector ${\boldsymbol{\varphi}}_{t}$. Having introduced the problem at hand, we detail an example formulation in the context of D2D mobile caching.

III Solution via Dual Descent

This section details the SDSD algorithm for solving ${(\mathbf{P}_{1})}$ in an online fashion. To this end, the basic assumptions are first stated (Sec. III-A), followed by the SDSD algorithm (Sec. III-B), and a discussion of the known results (Sec. III-C).

III-A Basic assumptions

The following assumptions are commonly utilized by different dual algorithms proposed in the literature. None of these assumptions are too restrictive, and they can be easily verified for most resource allocation problems of interest.

1. A1.

**Non-atomic probability density function: ** The random variable ${\boldsymbol{\varphi}}_{t}$ has a non-atomic probability density function (pdf). 2. A2.

**Slater’s condition: ** There exists strictly feasible ${(\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}})}$, i.e., ${\mathbb{E}\left[{\mathbf{s}_{t}}(\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}}_{t})\right]}{>}0$. 3. A3.

**Bounded subgradients: ** The function ${\mathbf{s}_{t}}(\cdot,\cdot)$ takes bounded values, i.e., there exists a constant $G<\infty$ such that $\left\|\mathbf{s}_{t}(\cdot,\cdot)\right\|\leq G$ for all ${t\in\mathbb{N}_{0}}$.

In (A1), for ${\boldsymbol{\varphi}}_{t}$ to have a non-atomic pdf, it should not have any point masses or delta functions. Note that this requirement is not restrictive for a number of applications arising in wireless communications; see e.g. [8]. The Slater’s condition is also not restrictive, since a strictly feasible resource allocation can often be found for most real-world problems; see Sec. V, [7, 8] for examples. Finally, the bound in (A3) also holds for most resource allocation problems where the functions ${\mathbf{s}_{t}}(\cdot,\cdot)$ represent natural quantities such as instantaneous rate (cf. (1b)), indicator function for channel outage [6], or household power consumption [39]. Having introduced the basic assumptions, we are ready to state the SDSD algorithm.

III-B The stochastic dual subgradient algorithm

Towards solving ${(\mathbf{P}_{1})}$, consider the more tractable dual formulation, which has a finite number of optimization variables. Introducing a dual variable ${\bm{\lambda}}\in{\mathbb{R}}^{K}_{+}$ corresponding to the constraint (3), the Lagrangian is given by

[TABLE]

where the constraints in (4) are kept implicit. The dual function is obtained by maximizing ${L}({{\bm{\lambda}},{\mathbf{x}},{\mathbf{p}}})$ subject to (4), that is,

[TABLE]

Finally, the dual problem of ${(\mathbf{P}_{1})}$ is given by

[TABLE]

In general, since ${(\mathbf{P}_{1})}$ may be non-convex, it holds that $\mathsf{D}\geq\mathsf{P}$. It was shown in [7, Prop. 6] however, that under (A1)-(A3), it holds that $\mathsf{P}=\mathsf{D}$. The proof utilizes the Lyapunov convexity theorem, and holds even if at least one element of ${\boldsymbol{\varphi}}_{t}$ has an absolutely continuous cumulative distribution function (cdf) [40]. It is remarked that Lyapunov convexity has previously yielded similar results in control theory [41], economics [42], and wireless communications [43, 5].

The result on zero duality gap legitimizes the dual descent approach, since the dual problem is always convex, and the resultant dual solution can be used for primal recovery. To this end, similar problems in various contexts have been solved via the classical dual descent algorithm [7, 5, 28, 6], wherein the primal updates utilize various sampling techniques.

This paper considers the ergodic stochastic optimization (ESO) algorithm proposed in [8] for a similar problem111The ESO algorithm is a stochastic dual subgradient descent algorithm applied to a resource allocation problem in [8].. Applied to ${(\mathbf{P}_{1})}$, the ESO algorithm starts with an arbitrary ${\bm{\lambda}}_{0}$, and utilizes the following iterations for $t\in\mathbb{N}_{0}$,

[TABLE]

Here, $\Pi_{t}:=\{{\mathbf{p}}_{{\boldsymbol{\varphi}}_{t}}\in{\mathbb{R}}^{p}|{\mathbf{p}}\in{\mathcal{P}}\}$ is the set of all legitimate values of the vector ${\mathbf{p}}_{{\boldsymbol{\varphi}}_{t}}$ and $\mathring{{\mathbf{p}}}$ denotes all feasible vectors ${\mathbf{p}}_{t}({\bm{\lambda}_{t}})$. The ESO algorithm is motivated from the fact that for any ${\bm{\lambda}}\in{\mathbb{R}}^{K}_{+}$, ${\mathbf{s}_{t}}({\mathbf{x}}_{t}({\bm{\lambda}}),{\mathbf{p}}_{t}({\bm{\lambda}}))$ is a stochastic subgradient of the dual function $g({\bm{\lambda}})$. Consequently, the updates in (9) amount to solving (8) via the SSD algorithm with a constant step-size. The use of a constant step size is motivated from classical short memory adaptive algorithms such as the least mean squares algorithm. As stated earlier, the constant step-size algorithms can even handle abrupt changes in the problem parameters, without being restarted.

This paper considers the projected variant of the SSD algorithm for dual updates. Specifically, the updates in (9b) are projected on to a compact set $\mathcal{L}\subset{\mathbb{R}}_{+}^{K}$, and take the following form

[TABLE]

where for any ${\bm{\lambda}}\in{\mathbb{R}}^{K}$,

[TABLE]

for $1\leq i\leq K$. In other words, large values of $[{{\bm{\lambda}}}]_{i}$ are truncated to ${\lambda_{\max}}$, where $\lambda_{\max}\gg\left\|{\bm{\lambda}^{\star}}\right\|_{\infty}$. Such a modification is already applicable to any practical implementation of the SSD algorithm, where ${\bm{\lambda}}_{t}$ is not allowed to take arbitrarily large values. Since ${\bm{\lambda}^{\star}}$ is not known in advance, a bound on $\left\|{\bm{\lambda}^{\star}}\right\|_{\infty}$ is derived in Appendix A using (A2). Consequently, the following rule can be used for choosing $\lambda_{\max}$ in practice:

[TABLE]

where $\tilde{{\bm{\lambda}}}\in{\mathbb{R}}^{K}_{+}$, $({\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}}})$ is a strictly feasible solution to ${(\mathbf{P}_{1})}$ (cf. (A2)), $G$ is the subgradient bound (cf. A3) and ${\chi(\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}})}:=\min_{1\leq k\leq K}\mathbb{E}\left[[{\mathbf{s}_{t}}({\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}}_{t}})]_{k}\right]$. In general, the quantity $\chi(\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}})$ may be calculated empirically. However, for many problems of interest, a bound on $\lambda_{\max}$ may arise naturally (cf. Sec. VI). The projected SSD proposed in (10) ensures that the iterates ${\bm{\lambda}}_{t}$ stay bounded for all ${t\in\mathbb{N}_{0}}$. The boundedness condition is required for carrying out the rate analysis in Sec. IV.

III-C Known results

The asymptotic properties of the SDSD algorithm with constant step-size are well-known [10, 11]. Asymptotic convergence results for the ESO algorithm, applied to slightly different resource allocation problem, were established in [8]. The results in [8] can readily be extended to ${(\mathbf{P}_{1})}$ solved via projected SDSD, and take following form:

[TABLE]

where the running average $\bar{{\mathbf{x}}}_{t}$ is defined as

$\bar{{\mathbf{x}}}_{{t}}=\frac{1}{{t}}\sum_{{\tau}=0}^{{t}-1}{\mathbf{x}}_{\tau}({\bm{\lambda}}_{\tau})$ and $\bar{G^{2}}$ is the bound on $\mathbb{E}\left[\left\|{\mathbf{s}_{t}}({\mathbf{x}}_{t}({\bm{\lambda}}_{t}),\!{\mathbf{p}}_{t}(\!{\bm{\lambda}}_{t}\!))\right\|^{2}\!\right]$. An important feature of the stochastic algorithm is that the primal updates in (9a) can be used for allocating resources in real-time. Further, such allocations will be asymptotically feasible and near-optimal for almost every realization of the random process $\{{\boldsymbol{\varphi}}_{t}\}_{{t\in{\mathbb{N}}_{0}}}$.

IV Convergence Rate Results

This section develops various results regarding the rate of convergence of the SDSD algorithm. In contrast to the asymptotic results in (13), the goal here is to quantify the rate at which the allocations specified by (9a) become optimal. Such results are of practical significance to the protocol designers, since they can be used to estimate the number of iterations required for the primal and dual objectives to be near-optimal. In the case of the constant step-size SDSD, the convergence rate also depends on the step-size parameter $\epsilon$. For instance, it is well-known that the choice $\epsilon\rightarrow 0$, motivated from the result in (13b), leads to slow convergence in all constant step-size (sub-)gradient descent algorithms. The results presented here provide a precise characterization of the trade-off between $\epsilon$ and the convergence rate for the SDSD algorithm.

As in [8], the results in this section make use of the strong law of large numbers, and thus hold for almost every realization of the i.i.d. process $\{{\boldsymbol{\varphi}}_{t}\}_{{t\in{\mathbb{N}}_{0}}}$. It is emphasized that the analysis presented here is quite different from the standard convergence analysis carried out for SSD algorithm and its variants [10, 11, 22, 24]. It is also different from the non-asymptotic analysis for the case of diminishing step-size SSD algorithms, that only applies to ensemble averages [15]. Furthermore, the rate results presented in [15] apply only to the unconstrained stochastic subgradient algorithm, and cannot be extended to constrained problems (cf. ${(\mathbf{P}_{1})}$) or to the projected subgradient algorithm (cf. (10)).

The results are first developed for the general SSD algorithm (Sec. IV-A), and subsequently specialized to the resource allocation problem at hand (Sec. IV-B).

IV-A Convergence rate for the SSD algorithm

This section considers the generic optimization problem

[TABLE]

where $\Lambda\subset{\mathbb{R}}^{K}$ is a closed, compact, and convex set, and $\max_{{\bm{\lambda}}\in\Lambda}\left\|{\bm{\lambda}}-{\bm{\lambda}}^{\star}\right\|\leq\Lambda_{\max}<\infty$. Similar to (8), the optimum function value is denoted by $\mathsf{D}=g({\bm{\lambda}}^{\star})$. Given ${\bm{\lambda}}\in\Lambda$, let ${\mathbf{f}}({\bm{\lambda}})\in\partial g({\bm{\lambda}})$ be a subgradient of $g({\bm{\lambda}})$. Similar to ${(\mathbf{P}_{1})}$, let ${\mathbf{f}_{t}}({\bm{\lambda}}):={\mathbf{f}}_{{\boldsymbol{\varphi}}_{t}}({\bm{\lambda}})$ for all ${t\in{\mathbb{N}}_{0}}$ be the corresponding stochastic subgradients that depend on the i.i.d. process $\{{\boldsymbol{\varphi}}_{t}\}_{{t\in{\mathbb{N}}_{0}}}$ and satisfy $\mathbb{E}\left[{\mathbf{f}_{t}}({\bm{\lambda}})\right]={\mathbf{f}}({\bm{\lambda}})$ for any ${\bm{\lambda}}\in\Lambda$. For instance, in the simplest case, the stochastic gradient could be of the form ${\mathbf{f}_{t}}({\bm{\lambda}})={\mathbf{f}}({\bm{\lambda}})+\boldsymbol{\zeta}_{t}$, where $\boldsymbol{\zeta}_{t}$ is a zero mean i.i.d. random variable. The optimization problem (14) is solved via the projected SSD algorithm.

[TABLE]

where $\mathcal{P}_{\Lambda}\left(\cdot\right)$ denotes the projection operation. The algorithm is initialized with an arbitrary ${\bm{\lambda}}_{0}\in\Lambda$ such that $B_{0}:=\left\|{\bm{\lambda}}_{0}-{\bm{\lambda}}^{\star}\right\|<\infty$. Next, we make certain assumptions specific to (14). To this end, define the stochastic error ${\mathbf{e}}_{t}({\bm{\lambda}}):={\mathbf{f}}_{t}({\bm{\lambda}})-{\mathbf{f}}({\bm{\lambda}})$, and observe that for any ${{\bm{\lambda}}\in\Lambda}$, the sequence $\{{\mathbf{e}}_{t}({\bm{\lambda}})\}$ is also i.i.d.

1. A3${}^{\prime}.$

**Bounded subgradients: ** There exists constant $G<\infty$ such that $\left\|{\mathbf{f}}_{t}({\bm{\lambda}})\right\|\leq G$ for all ${\bm{\lambda}}\in\Lambda$. 2. A4.

**Continuously differentiable error: ** The error function ${\mathbf{e}}_{t}({\bm{\lambda}})$ is continuously differentiable on $\Lambda$, and the gradient with respect to ${\bm{\lambda}}$ satisfies $\left\|\nabla_{{\bm{\lambda}}}{\mathbf{e}}_{t}({\bm{\lambda}})\right\|<G_{e}<\infty$.

The requirement for bounded subgradient in (A3′) is analogous to that in (A3) for ${(\mathbf{P}_{1})}$. Here, (A3′) is stated separately because the problem in (14) is more general than the dual of (2)-(4). In practice, applying the results of this section to (8) entails substituting ${\mathbf{f}}_{t}(\lambda)=\mathbf{s}_{t}({\mathbf{x}}({\bm{\lambda}}),{\mathbf{p}}_{t}({\bm{\lambda}})$, which makes (A3’) the same as (A3). The error function ${\mathbf{e}}_{t}({\bm{\lambda}})$ may not always be continuous or differentiable for the problem at hand, and the same must be verified explicitly. It is emphasized that (A4) need only be checked for ${\mathbf{e}}_{t}({\bm{\lambda}})$ and not for ${\mathbf{f}}_{t}({\bm{\lambda}})$, which is still allowed to be non-differentiable; see Sec. V.

As an example, consider the class of problems where the convex objective function takes the form $g({\bm{\lambda}})\!=\!\mathbb{E}\left[\ell_{t}({\bm{\lambda}})\right]\!+\!r({\bm{\lambda}})$, where $\ell_{t}(\!{\bm{\lambda}}\!)$ is a twice-differentiable loss function that depends on the ‘data index’ $t$, and $r({\bm{\lambda}})$ is a possibly non-differentiable regularizer. For such problems, the error function becomes ${\mathbf{e}}_{t}({\bm{\lambda}})\!\!=\!\!\nabla_{{\bm{\lambda}}}\ell_{t}({\bm{\lambda}})\!-\!\mathbb{E}\left[\nabla_{{\bm{\lambda}}}\ell_{t}({\bm{\lambda}})\right]$, which is clearly differentiable. Further, the ‘loss-plus-regularizer’ problem structure is quite general, and includes well-known formulations such as LASSO [24] and nuclear norm regularized matrix least squares [44]. Specifically, given regressands $\{\!y_{t}\!\}$ and regressors $\{\!{\mathbf{x}}_{t}\!\}$, the objective function in the LASSO formulation takes the form $\sum_{t}(y_{t}\!-\!\langle{\mathbf{x}}_{t},\!{\bm{\lambda}}\rangle)\!\!+\!\!\left\|{\bm{\lambda}}\right\|_{1}$ and thus adheres to (A4). The first result is regarding the objective function values obtained from (15), and holds for all $T$.

Theorem 1**.**

Under (A1)-(A4) and for $T=n/\epsilon$, the minimum dual function value is bounded as

[TABLE]

where the random variable $C_{T}(n,\epsilon)$ holds for $\zeta>0$,

[TABLE]

where $\nu:=\max\{\frac{1}{\epsilon},n\}$ and $A<\infty$ is a constant that does not depend on $n$ or $\epsilon$.

It is remarked that since $g(\cdot)$ is convex, the bound in (16) also holds for $g(\bar{{\bm{\lambda}}}_{t})$, where $\bar{{\bm{\lambda}}}_{t}:=\frac{1}{t}\sum_{\tau=0}^{t-1}{\bm{\lambda}}_{\tau}$ is the running average of the iterates. Theorem 1 characterizes the manner in which the minimum objective function value approaches $\mathsf{D}$ for large $t$. Of the three terms in this optimality gap, the first one depends on the initialization and decays as $\mathcal{O}(1/n)$. The second term depends on the subgradient bound, and decays linearly with the step-size $\epsilon$. Finally, the third term is random, and decays almost surely as $\mathcal{O}((\epsilon/n)^{1/2-\zeta})$ for any $\zeta>0$ (cf. (17)). Alternatively, the probability of the third term being non-zero decays exponentially as either $n\rightarrow\infty$ or $\epsilon\rightarrow 0$ (cf. (17c)). Indeed, for a given run of (15) with a fixed $\epsilon$, the probability of the third term being non-negligible starts to decrease only beyond $n>1/\epsilon$ or equivalently, $T>1/\epsilon^{2}$.

Further intuition on the convergence rate can be obtained by considering the two cases in (17). When $\epsilon>0$ is fixed, it can be seen that the asymptotic results in [10, 11, 8] follow directly from Theorem 1 as $n\rightarrow\infty$. That is, while the initial condition is “forgotten” for $t\gg 1/\epsilon$, the optimality gap does not necessarily approach zero, but is eventually bounded by $\epsilon G^{2}/2$. At the same time, the fluctuations due to the stochastic term subside exponentially fast; see (17c).

On the other extreme, consider the case when $n$ is kept fixed, while the algorithm is run for different values of $\epsilon$. For the scenarios when $\epsilon$ is arbitrarily small, the asymptotic optimality gap is clearly negligible. However, for such small step-sizes, the algorithm takes a long time to forget the initial conditions, since the first term decays only as $\mathcal{O}(\frac{1}{\epsilon t})$. Consequently, for all runs when $\epsilon$ is taken to be small, the algorithm will appear to converge slowly. Likewise, the probability of the stochastic term being non-negligible starts to decrease exponentially only for $T>1/\epsilon^{2}$ (cf. (17c)). It is remarked that such a trade-off also applies to the classical subgradient method [16], and the result in Theorem 1 can be viewed as its stochastic counterpart. It is remarked that the results in [16] can be readily obtained by taking expectation on both sides of (16) since we have that $\mathbb{E}\left[C_{t}(n,\epsilon)\right]\!=\!0$. Observe further that unlike the results in [13, 14, 23, 15, 45] that hold on an average, the almost sure results in (16) cannot be specified in terms of problem parameters alone. Indeed, while it holds that $C_{t}(n,{\epsilon})\leq 2G^{2}(\Lambda_{\max}+1)$, such a bound is not very useful in the present case, as compared to the stronger convergence rate result in (16).

Finally, it is remarked that it may be possible to minimize the bound in (16) to the extent possible, by fixing $T$ and choosing a corresponding step size. In the present case, given $T$, the bound is the smallest when $\epsilon=1/\sqrt{T}$ which yields the following result

[TABLE]

where the random variable $C_{T}=\mathcal{O}(T^{-1/4})$ almost surely. The result in Theorem 1 may therefore be seen as the generalization of the results in [24, 15] that have also reported an $\mathcal{O}(T^{-1/2})$ bound on average but have not analyzed the almost sure behavior. It is emphasized however that in practice, minimizing the bound may not necessarily translate to an improved convergence rate. Moreover, the number of iterations $T$ for which the algorithm runs may not necessarily be known in advance, e.g., in target tracking applications. Instead, it may be simpler to specify a fixed value of $\epsilon$, and continue to run the algorithm till the contribution of the $\mathcal{O}(\frac{1}{n})$ term becomes tolerably small.

Before proceeding with the proof of Theorem 1, an intermediate lemma establishing rate results on various time-averages is provided. The proof of Theorem 1 will subsequently utilize these results by expressing the optimality gap in (16) in terms of these time-averages.

Lemma 1**.**

Let ${\mathcal{T}}:=\{t_{1},t_{2},\ldots,t_{\left|{\mathcal{T}}\right|}\}$ be a set of natural numbers such that $t_{i}\neq t_{j}$. Then for any $T\geq\left|{\mathcal{T}}\right|$ and ${\bm{\lambda}},{\bm{\lambda}}^{\prime}\in\Lambda$, it holds under (A3′)-(A4) that

[TABLE]

where, for a given $\zeta>0$, the random variables $\{{L^{i}_{T}({\mathcal{T}})}\}_{i=1,2}$ satisfy

[TABLE]

where the constant $A_{i}<\infty$ does not depend on $T$.

Proof:

Observe that the i.i.d. process $\{{\mathbf{e}}_{t}({\bm{\lambda}})\}_{t\in{\mathcal{T}}}$ is zero-mean and satisfies

[TABLE]

where the last inequality holds from (A4). Therefore, it follows from the strong law of large numbers that for any $T\geq\left|{\mathcal{T}}\right|$, $\frac{1}{T}\sum_{t\in{\mathcal{T}}}{\mathbf{e}}_{t}({\bm{\lambda}})\rightarrow\mathbf{0}$ almost surely as $T\rightarrow\infty$. It can also be seen that the same holds for $L_{T}^{1}({\mathcal{T}}):=\left\|\frac{1}{T}\sum_{t\in{\mathcal{T}}}{\mathbf{e}}_{t}({\bm{\lambda}})\right\|$. The rate results in (20) hold as consequences of the strong law of large numbers for i.i.d. sequences with bounded moments; see [46, Chap. 7] for (20a). Finally, (20b), follows from the Bernstein inequality applied to i.i.d. zero-mean and bounded random vectors $\{e_{t}({\bm{\lambda}})\}$ [47].

Denote the $j$-th entry of ${\mathbf{e}}_{t}({\bm{\lambda}})$ by $e^{j}_{t}(\!{\bm{\lambda}}\!)$ for $1\!\leq\!j\!\leq K$. From (A3′)-(A4), we have that $e^{j}_{t}({\bm{\lambda}})$ is bounded and continuously differentiable on $\Lambda$. Consider arbitrary ${\bm{\lambda}}\!\neq\!{\bm{\lambda}}^{\prime}\!\in\!\Lambda$, and observe that since $\Lambda$ is convex, it holds for any $\beta\!\in\![0,\!1]$ that ${\bm{\lambda}}_{\beta}\!:=\!\beta{\bm{\lambda}}\!+\!(\!1\!-\!\beta){\bm{\lambda}}^{\prime}\!\in\!\Lambda$. It is now possible to use the mean-value theorem, which guarantees that there exists some $\beta_{j}\!\in\![0,\!1]$, such that

[TABLE]

Here, $\nabla{\mathbf{e}}^{j}_{t}({\bm{\lambda}}_{\beta_{j}})$ is an i.i.d. random variable that is also zero-mean, since for continuously differentiable and bounded functions (cf. (A4)), we have that $\mathbb{E}\left[\nabla{\mathbf{e}}^{j}_{t}({\bm{\lambda}}_{\beta_{j}})\right]=\nabla\mathbb{E}\left[e^{j}_{t}({\bm{\lambda}}_{\beta_{j}})\right]=0$. Taking summation in (22) and stacking the $K$ components, it follows for any $T\geq\left|{\mathcal{T}}\right|$, that

[TABLE]

where the $K\times K$ matrix ${\mathbf{E}}_{T}({\bm{\lambda}},{\bm{\lambda}}^{\prime})$ is defined as $[{\mathbf{E}}_{T}({\bm{\lambda}},{\bm{\lambda}}^{\prime})]_{jk}:={\frac{1}{T}\sum\limits_{t\in{\mathcal{T}}}}[\nabla e^{j}_{t}({\bm{\lambda}}_{\beta_{j}})]_{k}$, where the subscript is used to denote the $k$-th element of vector $\nabla e^{j}_{t}({\bm{\lambda}}_{\beta_{j}})$. Applying the Cauchy-Schwarz inequality to (23), we obtain $\left\|\frac{1}{T}\sum_{t\in{\mathcal{T}}}{\mathbf{e}}_{t}({\bm{\lambda}})-{\mathbf{e}}_{t}({\bm{\lambda}}^{\prime})\right\|\leq\left\|{\mathbf{E}}_{T}({\bm{\lambda}},{\bm{\lambda}}^{\prime})\right\|\left\|{\bm{\lambda}}-{\bm{\lambda}}^{\prime}\right\|$. From the strong law of large numbers, we have that $[{\mathbf{E}}_{T}({\bm{\lambda}},{\bm{\lambda}}^{\prime})]_{jk}\rightarrow 0$ almost surely as $T\rightarrow\infty$ for all $1\leq j,k\leq K$. It can be seen that the same also holds for $L^{2}_{T}({\mathcal{T}}):=\left\|{\mathbf{E}}_{T}({\bm{\lambda}},{\bm{\lambda}}^{\prime})\right\|$. Finally, the rate results in (20) follow from [46, Chap. 7]2 and the Bernstein inequality applied to i.i.d. zero-mean and bounded random variables $\{[\nabla e_{t}^{j}({\bm{\lambda}}_{\beta_{j}})]_{k}\}$ [47]. ∎

The proof of Theorem 1 follows in two steps: the derivation of the overall form required in (16), presented next; and the analysis of the random term $C_{t}(n,\epsilon)$ deferred to Appendix B.

Proof:

In order to derive the bound in (16), recall that since $g({\bm{\lambda}})$ is convex, we have that,

[TABLE]

Letting $g_{t}:=g({\bm{\lambda}_{t}})$, it follows from the non-expansive property of $\mathcal{P}_{\Lambda}\left(\cdot\right)$ that

[TABLE]

where (27) follows from (A3’) and (28) follows from (24). Rearranging (28) yields

[TABLE]

Taking sum over $\tau=0,1,\ldots,t-1$ and noting that $B_{0}=\delta_{0}$ and that $\epsilon t\geq n$, yields

[TABLE]

where the last inequality follows since $\delta_{t}\geq 0$ and the stochastic term in (30) is defined as

[TABLE]

Since (30) is of the same form as required in (16), it remains to show that $C_{t}(n,\epsilon)$ converges in the sense of (17)-(17c). The convergence analysis for $C_{t}(n,\epsilon)$ makes use of the bounds developed in Lemma 1 and is deferred to Appendix B. ∎

It is remarked that the results in Theorem 1 can likely be generalized to the case when $\Lambda$ is not necessarily compact. Such a generalization is likely possible because the strong law of large numbers, as well as the rate results in [46, Chap. 7] and [48, Theorem 1] only require the random process to have bounded moments. Nevertheless, the requirement that $\left\|{\bm{\lambda}}_{t}-{\bm{\lambda}^{\star}}\right\|\leq\Lambda_{\max}<\infty$ is not too restrictive, and greatly simplifies the analysis.

IV-B Convergence rate for the SDSD algorithm

In order to apply the results developed in Sec. IV-A to the dual problem (8), observe that the stochastic subgradient of $g({\bm{\lambda}})$ for any ${\bm{\lambda}}\in{\mathbb{R}}^{K}_{+}$ is given by

[TABLE]

With ${\mathbf{f}_{t}}({\bm{\lambda}})$ as defined in (32), the projected SSD updates take the same form as (15), with $\Lambda_{\max}=2\sqrt{K}\lambda_{\max}$. Further the bound required in (A3′) follows from (A3). Therefore, Theorem 1 applies as is to the dual objective function under (A3) and (A4).

For the resource allocation problem however, the behavior of the primal objective function is more important. The subsequent theorem characterizes the primal near-optimality when the running average of $\{{\mathbf{x}}_{t}({\bm{\lambda}}_{t})\}$ is used for allocating resources. For the purpose of rate analysis, time is divided into epochs of duration $\frac{1}{\epsilon}$ each, and the result is expressed in terms of $\epsilon$ and $n$.

Theorem 2**.**

Under (A1)-(A4), and for $n/\epsilon\leq t<(n+1)/\epsilon$, the average primal objective function is near optimal in the following sense:

[TABLE]

where $R_{0}:=\left\|{\bm{\lambda}}_{0}\right\|^{2}$, $\bar{{\mathbf{x}}}_{t}=\frac{1}{t}\sum_{\tau=0}^{t-1}{\mathbf{x}}_{\tau}$, and the random variable ${C^{\prime}_{t}(n,\epsilon)}$ is such that for $\zeta>0$,

[TABLE]

where $\nu:=\max\{\frac{1}{\epsilon},n\}$ and $A<\infty$ is a constant that does not depend on $n$ or $\epsilon$.

The term $C^{\prime}_{t}(n,\epsilon)$ in Theorem 2 is very similar to $C_{t}(n,\epsilon)$ in Theorem 1, and therefore decays at the same rate. It follows from Theorem 2 that the resource allocation yielded by the projected SDSD algorithm is near optimal since the average primal objective value is close to $\mathsf{P}$. Similar to (16), the bound in (35) also holds for ${\max_{0\leq\tau\leq t-1}f_{0}({\mathbf{x}}_{\tau})}$, as well as for $f_{0}(\bar{{\mathbf{x}}}_{t})$. Further, the optimality gap in (35) is also similar to the one in (16), and therefore decays at the same rate. For details, see the discussion after the statement of Theorem 1.

In order to prove Theorem 2, the specific form of the bound in (35) is first established. The rest of the proof is much the same as before, and results from Lemma 1 are again used to derive the bounds on $C^{\prime}_{t}(n,\epsilon)$ as in the proof of Theorem 1.

Proof:

Recall that the subgradient of $g({\bm{\lambda}})$ is given by ${\mathbf{f}}({\bm{\lambda}})=\mathbb{E}\left[{\mathbf{s}_{t}}({\mathbf{x}}_{t}({\bm{\lambda}}),{\mathbf{p}}_{t}({\bm{\lambda}}))\right]$, so that $g({\bm{\lambda}})=f_{0}({\mathbf{x}}({\bm{\lambda}}))+{\langle{\bm{\lambda}},{\mathbf{f}}({\bm{\lambda}})\rangle}$. Since $f_{0}$ is concave, the following inequalities hold:

[TABLE]

Next, the second term in (38) can be bounded as

[TABLE]

where, (39) follows from the non-expansiveness property of the projection operator and from the fact that $\boldsymbol{0}\in\Lambda$ $\left(\left\|{\bm{\lambda}}_{t+1}\right\|=\left\|{\bm{\lambda}}_{t+1}-\boldsymbol{0}\right\|\right)$, and (41) follows from (A3). Taking sum over $\tau=0,1,\ldots,t-1$ and dividing by $2\epsilon t$ yields

[TABLE]

where, $C^{\prime}_{t}(n,\epsilon):=\left|\frac{1}{t}\sum_{\tau=0}^{t-1}{\langle{\mathbf{f}}_{\tau}({\bm{\lambda}}_{\tau})-{\mathbf{f}}({\bm{\lambda}}_{\tau}),{\bm{\lambda}}_{\tau}\rangle}\right|.$ The bound in (35) follows by plugging back (42) into (38). The analysis for $C^{\prime}_{t}(n,\epsilon)$ is much the same as in the proof of Theorem 1. The only difference for this case is that the iterate bound becomes $\left\|{\bm{\lambda}}_{t}\right\|\leq\sqrt{K}\lambda_{\max}$ from (10). Consequently, after rearranging various terms in $C^{\prime}_{t}(n,\epsilon)$ and using the triangle inequality in (69), (82), and (83), all occurrences of $\Lambda_{\max}$ get replaced with $\sqrt{K}\lambda_{\max}$. Since this is equivalent to redefining the constant $\Lambda_{\max}$ appropriately, the required rate results continue to hold. ∎

V Application to D2D Communications

This section details some implementation aspects of the SDSD algorithm in the context of D2D communication problem considered in this paper under slow and fast fading scenarios. The Assumptions (A1)-(A4) are also verified for the problems at hand so as to ensure that Theorems 1 and 2 hold.

Before proceeding, the SDSD algorithm for the general form of the D2D problem (1) is detailed. Specifically, the Lagrangian is given by

[TABLE]

which yields the following stochastic algorithm. Since the Lagrangian is separable in $r$ and $p^{i}_{t}$, starting with arbitrary $\lambda_{0}$, the primal iterates at time slot $t$ become:

[TABLE]

At the end of each time slot, the dual variable is updated as

[TABLE]

Recall that the set of functions $\mathcal{P}$ is such that only one user, denoted by

[TABLE]

is allocated non-zero power at time slot $t$. Therefore, the dual variable is updated as

[TABLE]

The full algorithm is summarized in Algorithm 1.

The rate analysis developed in Sec. IV applies to the present problem under the following assumptions.

1. B1.

*Continuous random variables: * The random variables ${\boldsymbol{\varphi}}_{t}=({\mathcal{M}_{t}},\{c^{i}_{t}\}_{{i\in\mathcal{M}_{t}}},\{\gamma^{i}_{t}\}_{{i\in\mathcal{M}_{t}}})$ are i.i.d., have continuous cdfs, and finite supports, i.e., ${\mathcal{M}_{t}}\subset\mathcal{M}$, $\gamma_{t}^{i}\in[\gamma_{\min},\gamma_{\max}]$, and $c^{i}_{t}\in[c_{\min},c_{\max}]$ for each ${i\in\mathcal{M}_{t}}$. 2. B2.

Power constraints: The set $\mathcal{P}:=\{{\mathbf{p}}:{\mathbb{R}}^{3M}\rightarrow{\mathbb{R}}^{M}\mid{\mathbf{p}}_{{\boldsymbol{\varphi}}}\in\Pi_{{\boldsymbol{\varphi}}}\}$, where for any ${\boldsymbol{\varphi}}_{t}$, we have that $\Pi_{{\boldsymbol{\varphi}}_{t}}:={\Pi_{t}}=\{{\mathbf{p}}_{t}\in{\mathbb{R}}^{M}\mid p_{t}^{j}=0~{}{j\notin{\mathcal{M}_{t}}},\left\|{\mathbf{p}}_{t}\right\|_{0}=1,p^{i_{t}}_{t}c_{t}^{i_{t}}\in[C_{\min},C_{\max}]\}$, where $i_{t}:=\arg\max_{{i\in\mathcal{M}_{t}}}p^{i}_{t}$ and $p^{i}_{t}=[{\mathbf{p}}_{t}]_{i}$. 3. B3.

*High SNR: * It is assumed that $\gamma^{i}_{t}\gg 1$ for all ${i\in\mathcal{M}_{t}}$.

The finite support of the random quantities is again motivated from practical considerations. The set $\mathcal{P}$ also includes limits on the maximum affordable cost $C_{\max}$ and the minimum operational cost or minimum allowable transaction amount $C_{\min}$. A maximum power constraint of the form $p^{i}_{t}\leq P_{\max}$ may also be included within $\mathcal{P}$. However, for the present application, it is assumed that the caches are not energy constrained, so that $P_{\max}\gg C_{\max}/c^{i}_{t}$ for all ${i\in\mathcal{M}_{t}}$. In other words, the user’s cost constraint is much more stringent than the cache’s energy constraint. Finally, the high SNR assumption is justified if there are always enough mobile caches available at all slots. In a typical setting, the MoUE may “see” hundreds of advertisements from potential mobile cache servers, but may choose to consider only tens of users with which control messages may be exchanged easily. Next, the discussion for slow and fast fading cases will be carried out.

V-A Slow Fading

Recall that under slow fading, since power allocation occurs every coherence interval, we have for high SNR (cf. (B3)), that R_{i}(p_{t}^{i}(\lambda_{t}),\gamma_{t}^{i}):\approx W{\color[rgb]{0,0,0}\log_{2}}(p_{t}^{i}(\lambda_{t})\gamma_{t}^{i}/\alpha). The primal iterate in (45) can be found in two steps. First the optimum transmit power for all potential users is determined, i.e., for each ${i\in\mathcal{M}_{t}}$,

[TABLE]

The winning user is the one that maximizes the objective function, i.e.,

[TABLE]

where the expression in (51) derived in Appendix C. An implication of (51) is that the random variable $i_{t}$ is i.i.d. Finally, it holds that $p^{j}_{t}=\hat{p}^{i_{t}}_{t}(\lambda_{t})$ for $j=i_{t}$ and zero otherwise. Similarly, $r_{t}$ is calculated as ${r}_{t}(\lambda_{t})=\left[\frac{1}{\lambda_{t}}\right]_{r_{\min}}^{r_{\max}}$ resulting in the dual update

[TABLE]

An additional assumption regarding the parameter values is made in the slow fading case:

B4.   *Strict feasibility: * The problem parameters satisfy r_{\min}<\mathbb{E}\left[\max_{i}{\color[rgb]{0,0,0}\log_{2}}(C_{\max}\gamma_{t}^{i}/c^{i}_{t})\right].

The strict feasibility condition is required for ensuring the existence of a Slater point. Since it holds that ${\gamma_{t}^{i}/c^{i}_{t}\geq{\gamma_{\min}/c_{\max}}}$, it is possible to satisfy (B4) by keeping $r_{\min}$ sufficiently small and/or if $\gamma_{\min}$ is sufficiently large.

Having stated the algorithm and all required assumptions, the following Lemma summarizes the main result of this subsection.

Corollary 1**.**

Under (B1)-(B4), the iterates obtained from (50)-(52) adhere to the rate bounds stated in Theorems 1 and 2.

Proof:

For the results in Theorem 1 and 2 to apply, it suffices to verify that assumptions (A1)-(A4) are satisfied under the slow fading case. The random variable ${\boldsymbol{\varphi}}_{t}$ has a non-atomic pdf since the channel gains $\gamma^{i}_{t}$ have a continuous cdf, thus confirming (A1); see also [40]. The Slater’s condition is met by choosing $\tilde{r}=r_{\min}$ and $\tilde{p}^{i_{t}}_{t}=C_{\max}/c_{t}^{i_{t}}$ where $i_{t}$ is given in (51) and zero for all $j\neq i_{t}$. For such a choice, it holds from (B4) that \tilde{r}<\mathbb{E}\left[{\color[rgb]{0,0,0}\log_{2}}(\tilde{p}^{i_{t}}_{t}\gamma^{i_{t}}_{t})\right], which is the required condition for strict feasibility. For a given $\lambda$, the subgradient function is given by

[TABLE]

where $i_{t}$ and $p^{i_{t}}_{t}$ are evaluated as in (51) and (50). A bound on the subgradient (cf. (A3)) may therefore be found as \left|f_{t}(\lambda)\right|\leq W{\color[rgb]{0,0,0}\log_{2}}\left(\frac{C_{\max}\gamma_{\max}}{\alpha c_{\min}}\right)+r_{\max}=:G. Next, in order to verify (A4), the expression for the stochastic subgradient error $e_{t}(\lambda):=f_{t}(\lambda)-\mathbb{E}\left[f_{t}(\lambda)\right]$ is first derived. Recalling that $i_{t}=\operatorname*{arg\,max}_{i}\gamma^{i}_{t}/c^{i}_{t}$, consider the following three cases,

When $\lambda<C_{\min}/W$, it holds that $p^{i_{t}}_{t}=C_{\min}/c^{i_{t}}_{t}$, implying that

[TABLE]

where the expectations are with respect to ${\boldsymbol{\varphi}}_{t}$. 2. 2.

When $C_{\min}/W\leq\lambda\leq C_{\max}/W$, it holds that $p^{i_{t}}_{t}=W\lambda\gamma^{i_{t}}_{t}/c^{i_{t}}_{t}$, implying that

[TABLE] 3. 3.

Similarly, when $\lambda>C_{\max}/W$, it holds that $p^{i_{t}}_{t}=C_{\max}/c^{i_{t}}_{t}$, implying that

[TABLE]

Therefore, the subgradient error is a zero-mean random variable that does not depend on $\lambda$, and is therefore trivially continuously differentiable in $\lambda$. ∎

V-B Fast Fading

In the more realistic fast fading case, the power allocation and downloads occur over several coherence intervals. Under the high SNR assumption, the rate becomes , R_{i}(p^{i}_{t},\gamma^{i}_{t})\approx W{\color[rgb]{0,0,0}\log_{2}}(p_{t}^{i}\gamma_{t}^{i}/\alpha)+W\psi_{i} where \psi_{i}=\mathbb{E}_{h}\left[{\color[rgb]{0,0,0}\log_{2}}(h_{i})\right] for a given user $i$ [38]. As in the slow fading case, the primal iterates are again found in two steps. First, the power allocation for a potential user $i$ is found,

[TABLE]

It is shown in Appendix C that the winning user for the fast fading case can be written as

[TABLE]

Finally, since $r_{t}(\lambda_{t})={\max\{\min\{1/\lambda_{t},r_{\max}\},r_{\min}\}}$ as before, the dual update is given by

[TABLE]

In order to apply the rate bounds in Theorems 1 and 2, we again assume (B1)-(B3), and make the following assumption analogous to (B4).

B5.   *Strict feasibility: * The problem parameters satisfy r_{\min}<\mathbb{E}\left[\max_{i}\left\{{\color[rgb]{0,0,0}\log_{2}}(C_{\max}\gamma_{t}^{i}/c^{i}_{t})+\psi_{i}\right\}\right].

As in the slow fading case, (B5) allows us to obtain a Slater point, as required by (A2). The following Lemma summarizes the result for the fast fading case.

Corollary 2**.**

Under (B1)-(B3) and (B5), the iterates obtained from (59)-(61) adhere to the rate bounds stated in Theorems 1 and 2.

Proof:

The As in Lemma 1, it suffices to verify assumptions (A1)-(A4). The random variable ${\boldsymbol{\varphi}}_{t}$ has a non-atomic pdf as remarked earlier. Similarly, it can be verified that the Slater point is given by $\tilde{r}=r_{\min}$ and $\tilde{p}^{i_{t}}_{t}=C_{\max}/c_{t}^{i_{t}}$ where $i_{t}$ is as given in (60), and zero for all $j\neq i_{t}$. The subgradient bound required in (A3) now becomes, \left|f_{t}(\lambda)\right|\leq W{\color[rgb]{0,0,0}\log_{2}}\left(\frac{C_{\max}\gamma_{\max}}{\alpha c_{\min}}\right)+W\psi_{\max}+r_{\max}=:G where $\psi_{\max}:=\max_{i}\psi_{i}$. Finally, in order to verify (A4), we proceed as in the proof of Lemma 1 and derive an expression for the subgradient error $e_{t}(\lambda):=f_{t}(\lambda)-\mathbb{E}\left[f_{t}(\lambda)\right]$. Since expression for the allocated power is the same for the two cases, it can be seen that for the fast fading case as well e_{t}(\lambda)=W{\color[rgb]{0,0,0}\log_{2}}\left(\frac{\gamma^{i_{t}}_{t}}{c^{i_{t}}_{t}}\right)-\mathbb{E}\left[W{\color[rgb]{0,0,0}\log_{2}}\left(\frac{\gamma^{i_{t}}_{t}}{c^{i_{t}}_{t}}\right)\right] where $i_{t}$ is found as in (60). Since $e_{t}(\lambda)$ does not depend on $\lambda$, (A4) also holds trivially in the fast fading scenario. ∎

VI Numerical Tests

This section describes the numerical tests on the D2D example discussed in Sec. V. The convergence rate of the SDSD algorithm is studied for the fast fading scenario depicted in Fig. 2. For the simulations, we consider $M\!=\!25$ operational UEs. At each time slot, the MoUE receives advertisements from a random subset $\mathcal{M}_{t}$ of 5 to 25 UEs. Without loss of generality, downloading from the $i$-th UE incurs a cost of $c^{i}_{t}\!=\!i$ per unit of transmit power. The lower and upper limits for each transaction are set as $C_{\min}=1$ and $C_{\max}=25$, respectively. The average channel gains $\gamma^{i}_{t}$ are assumed to be Rayleigh distributed with $\gamma_{\min}=0.1$ and $\gamma_{\max}=65$, and for simplicity, the parameters $\alpha$, and $\psi_{i}$ are all set to unity. In order to keep the numbers realistic, we set $W=1$ MHz. Finally, in order to ensure Slater’s condition, we set $r_{\min}=0.2$ and $r_{\max}=10$. In realistic scenarios, since the optimal rate is expected to be greater than $r_{\min}$, it follows from the definition of $r_{t}(\lambda_{t})$ in Sec.V-A that $\lambda^{\star}>1/r_{\min}$. Therefore it is safe to take $\lambda_{\max}\gg 1/r_{\min}$.

Fig. 2(a) shows the evolution of the utility function calculated using running averages $\bar{r}_{t}:=1/t\sum_{\tau=0}^{t-1}r_{\tau}$ of the allocated rate with iterations. As expected from Theorem 2, the utility function converges to a value that is closer to the optimal when $\epsilon$ is small. Similarly, Fig. 2(b) shows the evolution of the dual objective function, which again converges to a point closer to the optimal when $\epsilon$ is small. Observe from the results that for $\epsilon=0.1$, the oscillations continue even as number of iterations go to infinity, as implied by Theorem 1. These oscillations are allowed due to the presence of an $\mathcal{O}({\epsilon})$ term on the right-hand side of (17), and are well-documented for the constant step size stochastic subgradient type algorithms [13, 10, 8].

The convergence rate result of Theorem 1 is further illustrated in Fig. 3(a) and Fig. 3(b), where the deterministic terms in (16) are not included. The stochastic term $C_{t}(n,\epsilon)$ is calculated from (31) and plotted against both $\epsilon$ and $n$. It can be seen from both plots that $C_{t}(n,\epsilon)\rightarrow 0$ as either $n\rightarrow\infty$ or $\epsilon\rightarrow 0$, as claimed in Theorem 1.

Having studied the convergence properties of the SDSD algorithm, we now focus on some of the nuances of the edge-caching formulation in (1). To begin with, the performance of the proposed scheme is compared against that obtained from two naive algorithms: random and opportunistic. The maximum transmit power for the three cases is scaled so as to ensure equal aggregate power consumption. As the name suggests, an MoUE following the random scheme selects an available UE randomly and without paying any attention to the channel or the cost of the UE. The data is transmitted at the maximum power so as to ensure the maximum rate. As evident from Table I, such a scheme is able to obtain a higher download rate but also at a significantly higher cost. In contrast, the opportunistic scheme advocates a parsimonious approach wherein the MoUE always selects an available UE with the lowest cost. Subsequently, the UE transmits with the maximum power but ultimately achieves a lower aggregate download rate, due to suboptimal channel conditions.

Fig. 4 provides results from the perspective of the UEs and is generated by running the same algorithm for 1000 independent identically distributed MoUEs. In particular, if an MoUE follows the optimal policy determined by (1), the UEs may be interested in knowing a reasonable price value. As expected, it is clear from Fig. 4, that the UEs that charge more are selected less often have lower data usage. Consequently, the aggregate revenue of the UEs with the lowest charges is also the highest. More interestingly however, such high-priced UEs have a very high revenue per Mb of data served. The intuition here is that UEs with high costs are only selected when their channel gains are proportionally higher than the others. Therefore, all transmissions to such UEs occur at higher rates and correspondingly lower power. In summary, by operating only under favorable channel conditions, the high-priced UEs extract a greater revenue for every bit that they serve. Note however that the revenue appears to saturate, and increases very slowly for very high prices.

VII Conclusions

This paper considers a general stochastic resource allocation problem and solved using constant step-size stochastic subgradient descent algorithm in an online manner. A stochastic bound on the gap between the objective function and the optimum is developed and analyzed in an almost sure sense, generalizing the existing results. The bounds characterize the precise manner in which the optimality gap behaves for fixed and arbitrarily small step-sizes. The convergence rate analysis is also extended to a class of stochastic resource allocation problems that utilize stochastic dual subgradient descent (SDSD) iterations. Existing results on near-optimality of the primal average objective function are again generalized for convergence rate analysis. As an example, a resource allocation problem is formulated in the context of mobile caching in device-to-device communications, and solved via SDSD. The regularity conditions required for the rate analysis are verified, and numerical tests are provided, further substantiating the convergence rate results.

Appendix A A bound on $\left\|{\bm{\lambda}^{\star}}\right\|_{\infty}$

From (A2), there exists some $\tilde{{\mathbf{x}}}\in\mathcal{X}$ and $\tilde{{\mathbf{p}}}\in{\mathcal{P}}$, such that $\mathbb{E}\left[{\mathbf{s}_{t}}(\tilde{{\mathbf{x}}},{\tilde{{\mathbf{p}}}_{t}})\right]>0$, where recall that $\tilde{{\mathbf{p}}}_{t}:=\tilde{{\mathbf{p}}}_{{\boldsymbol{\varphi}}_{t}}$ for all ${t\in{\mathbb{N}}_{0}}$ and the expectation is with respect to ${\boldsymbol{\varphi}}_{t}$. Given ${\tilde{\bm{\lambda}}}\in{\mathbb{R}}^{K}_{+}$, define the sublevel set ${\mathbf{Q}}_{{\tilde{\bm{\lambda}}}}:=\{{\bm{\lambda}}\in{\mathbb{R}}^{K}_{+}\mid g({\bm{\lambda}})\leq g({\tilde{\bm{\lambda}}})\}$, and observe that for any ${\bm{\lambda}}\in{\mathbf{Q}}_{{\tilde{\bm{\lambda}}}}$, it holds that

[TABLE]

Rearranging the expression in (62), we obtain

[TABLE]

where ${\chi(\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}})}:=\min_{1\leq k\leq K}\mathbb{E}\left[\left[{\mathbf{s}_{t}}({\tilde{{\mathbf{x}}},\tilde{{\mathbf{p}}}_{t}})\right]_{k}\right]$. Observe that ${\mathbf{Q}}_{{\bm{\lambda}^{\star}}}=\{{\bm{\lambda}}\in{\mathbb{R}}^{K}_{+}\mid g({\bm{\lambda}})\leq\mathsf{D}\}$, so that it follows from (63) that

[TABLE]

Finally, since $g({\tilde{\bm{\lambda}}})\geq\mathsf{D}$ for all ${\tilde{\bm{\lambda}}}\in{\mathbb{R}}^{K}_{+}$, the bound in (64) can be relaxed to yield (12).

Appendix B Asymptotic properties of $C_{T}$

Proof:

In order to study the convergence rate of $C_{T}$, the time is divided into epochs of duration $1/\epsilon$, so that there exists some $n\!\geq\!0$ that satisfies $n/\epsilon\!\leq\!T\!<\!(n+1)/\epsilon$. Since $n:=\lfloor\epsilon t\rfloor$, where $\lfloor\cdot\rfloor$ denotes the floor operation, is an arbitrary number, such a split allows the value of $t$ to be increased by keeping either $n$ or $\epsilon$ fixed, and varying the other. It is therefore possible to separately study the effects of choosing larger $n$ or smaller $\epsilon$ values. For instance, if $\epsilon=0.1$, the time is divided into epochs of duration 10 iterations each. Hence, the zeroth epoch consists of iterations $0\leq t\leq 9$, the first epoch consists of iterations $10\leq t<19$, and so on. With such a split, the classical asymptotic analysis for $t\rightarrow\infty$ is equivalent to fixing $\epsilon$ and letting $n\rightarrow\infty$. Additionally, the proposed split allows us to study the case when $n$ is fixed, but the algorithm is run with different values of $\epsilon$.

This proof is devoted to the analysis of $C_{t}(n,\epsilon)$, and relies on rearranging the terms in (31) so that the results in developed in Lemma 1 can be applied. The proof is divided into two parts, one for each mode of convergence in (17).

Fixed $n\!<\!\infty$ and $\epsilon\rightarrow 0$: For this case, $C_{t}(n,\epsilon)$ is split into summands corresponding to each epoch till time $t$, that is,

[TABLE]

The limits in the summation are defined as $\ell_{m}:=m/\epsilon$ and $u_{m}:=\frac{(m+1)}{\epsilon}-1$ for $m<n$ while $u_{n}:=t-1$. Next, define for all ${\bm{\lambda}}\in\Lambda$ and $\ell_{m}\leq\tau\leq u_{m}$

[TABLE]

Substituting (66) in (65), we obtain

[TABLE]

Such a split allows us to use (19) in order to bound the magnitude of each term separately. Specifically, letting ${\mathcal{T}}_{m}:=\{\ell_{m},\ell_{m}+1,\ldots,u_{m}\}$,

[TABLE]

Similarly, denoting ${\mathcal{T}}^{\prime}_{\tau}:=\{\ell_{m},\ell_{m}+1,\ldots,\tau\}$ for all $\ell_{m}\leq\tau\leq u_{m}$, it holds from using triangle inequality and (19), that

[TABLE]

where the (71) uses the non-expansive property of the projection operator $\mathcal{P}_{\Lambda}\left(\cdot\right)$ and the boundedness of the stochastic subgradients (cf. (A3′)). Substituting (69) and (71) into the expression for $C^{m}(\epsilon)$ yields the following bound

[TABLE]

Finally, the bound for $C_{t}(n,\epsilon)$ becomes

[TABLE]

Therefore, the rate result from Lemma 1 implies that

[TABLE]

which goes to zero almost surely as $\epsilon\rightarrow 0$, yielding the bound in (17a). Likewise, let $A_{1m}<\infty$ be the constant associated with the bounds for $\Lambda_{\max}L_{1/\epsilon}^{1}({\mathcal{T}}_{m})$, as necessitated by Lemma 1. Then, using the union bound, it follows that

[TABLE]

where $A_{1}:=\sum_{m}A_{1m}$. Along the same lines, the result in Lemma 1 and the subsequent use of the union bound imply that there exist a constant $A_{2}<\infty$ such that the probability of the second term in (73) exceeds $\epsilon^{1/2-\zeta}$ is bounded by $A_{1}\exp\left(-\epsilon^{-2\zeta}\right)$. Combining the two bounds, and again using union bound, we have that

[TABLE]

where $A_{3}=A_{1}+A_{2}$.

*Fixed $\epsilon>0$ and $n\rightarrow\infty$: * In this case, $C_{t}(n,\epsilon)$ must now be split into two terms as follows,

[TABLE]

where,

[TABLE]

For this analysis, it is assumed without loss of generality that $1/\epsilon$ is an integer. That way, the subscripts $\frac{m}{\epsilon}+\tau$ are also integers and the floor operation is not required. Given $\epsilon$, note that $D(n)$ is a sum of a fixed number of bounded random variables, so that $D(n)/n\rightarrow 0$ surely as $n\rightarrow\infty$. In order to bound $C(n)$, define for all ${\bm{\lambda}}\in\Lambda$ and $0\leq\tau\leq 1/\epsilon-1$,

[TABLE]

Then, it follows that

[TABLE]

It is now possible to bound each term in (81) separately. Defining ${\mathcal{T}}^{\tau}:=\{m/\epsilon+\tau\}_{m=0}^{n-1}$, and using (19), it follows that

[TABLE]

Proceeding similarly,

[TABLE]

Finally, substituting (82) and (84) into the expression for $C(n)$, and noting that $1/\epsilon$ is a fixed number, the following bound is obtained

[TABLE]

[TABLE]

which goes to zero almost surely as $n\rightarrow\infty$, implying that $C_{t}(n,\epsilon)\rightarrow 0$ almost surely as $n\rightarrow\infty$. Both the rate results can again be inferred as in the previous case. Indeed, similar to (76), given $\zeta>0$, there exist $A_{4}<\infty$ such that $\mathbb{P}\left(C_{t}(n,\epsilon)>n^{\zeta-1/2}\right)\leq A_{4}\exp(-n^{2\zeta})$. Combining with (76), the probability bounds can be written as

[TABLE]

where $A\!\!=\!\max\{A_{3},\!A_{4}\}$. The required result follows by choosing $\nu=\max\{\frac{1}{\epsilon},n\}$ in (86). ∎

Appendix C Derivation of (51) and (60)

Consider first the slow fading case, where the winning user is given by

[TABLE]

where $\hat{p}^{i}_{t}$ is given by (50). Thus, the objective function in (87) for a given $\lambda$ can be written as

[TABLE]

Since {\color[rgb]{0,0,0}\log_{2}} is monotonic function, observe in (88) that in all three cases, $T^{i}_{t}(\lambda)$ depends monotonically on $\gamma^{i}_{t}/c^{i}_{t}$ for all $\lambda>0$. This allows us to conclude that $i_{t}=\operatorname*{arg\,max}_{{i\in\mathcal{M}_{t}}}T^{i}_{t}(\lambda)=\operatorname*{arg\,max}_{{i\in\mathcal{M}_{t}}}\frac{\gamma^{i}_{t}}{c^{i}_{t}}$ which is the required identity in (51). Similarly for the fast fading case, the objective function for the winning user in (60) is given by

[TABLE]

which, for $\lambda>0$, again depends monotonically on {\color[rgb]{0,0,0}\log_{2}}(\gamma^{i}_{t}/c^{i}_{t})+\psi_{i}. The expression in (60) therefore follows.