Model-Free Unsupervised Learning for Optimization Problems with Constraints

TL;DR

This paper introduces a model-free neural network-based framework for solving constrained optimization problems in wireless communications without requiring explicit models or optimal solutions, demonstrating efficiency through power control simulations.

Contribution

It presents a novel, model-free learning approach that uses neural networks to optimize functions and Lagrange multipliers without supervision, applicable to complex wireless problems.

Findings

Framework effectively solves power control problems.

Neural networks learn constraints and objectives simultaneously.

Demonstrates efficiency over traditional methods.

Abstract

In many optimization problems in wireless communications, the expressions of objective function or constraints are hard or even impossible to derive, which makes the solutions difficult to find. In this paper, we propose a model-free learning framework to solve constrained optimization problems without the supervision of the optimal solution. Neural networks are used respectively for parameterizing the function to be optimized, parameterizing the Lagrange multiplier associated with instantaneous constraints, and approximating the unknown objective function or constraints. We provide learning algorithms to train all the neural networks simultaneously, and reveal the connections of the proposed framework with reinforcement learning. Numerical and simulation results validate the proposed framework and demonstrate the efficiency of model-free learning by taking power control problem as an example.

Equations

$$P1:\underset{\mathbf{f}(\mathbf{h})}{\max }$$

$$s.t.$$

$${\mathbb{E}}_{\mathbf{h}}[\mathbf{c}(\mathbf{f}(\mathbf{h}),\mathbf{h})]⪯0$$

$$\mathcal{L}(\mathbf{f}(\mathbf{h}),\mathbf{\lambda }(\mathbf{h}),\mathbf{\xi })$$

$$={\mathbb{E}}_{\mathbf{h}}\left[J(\mathbf{f}(\mathbf{h}),\mathbf{h})-\mathbf{\lambda }(\mathbf{h}{)}^T\mathbf{g}(\mathbf{f}(\mathbf{h}),\mathbf{h})-{\mathbf{\xi }}^T\mathbf{c}(\mathbf{f}(\mathbf{h}),\mathbf{h})\right]$$

$$P2:\underset{\mathbf{\lambda }(\mathbf{h}),\mathbf{\xi }}{\min }\underset{\mathbf{f}(\mathbf{h})}{\max }$$

$$s.t.$$

$$\mathbf{\xi }⪰0$$

$$P3:\underset{{\mathbf{\theta }}_{\lambda },\mathbf{\xi }}{\min }\underset{{\mathbf{\theta }}_f}{\max }$$

$$s.t.$$

$$\mathbf{\xi }⪰0$$

$${\nabla }_{{\mathbf{\theta }}_f}\mathcal{L}=$$

$$\displaystyle-\nabla_{\mathbf{x}}\mathbf{c}(\mathbf{x},\mathbf{h})\bm{\xi}\big{]}\big{|}_{\mathbf{x}=\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})}\Big{]}$$

$${\nabla }_{{\mathbf{\theta }}_{\lambda }}\mathcal{L}=$$

$${\nabla }_{\mathbf{\xi }}\mathcal{L}=$$

$${\mathbf{\theta }}_f^{(t+1)}\leftarrow$$

$$\displaystyle-\nabla_{{\mathbf{x}}}\mathbf{g}(\mathbf{x},\mathbf{h})\tilde{\bm{\lambda}}(\mathbf{h};\bm{\theta}_{\lambda}^{(t)})-\nabla_{\mathbf{x}}\mathbf{c}(\mathbf{x},\mathbf{h})\bm{\xi}^{(t)}\Big{]}\Big{|}_{\mathbf{x}=\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f}^{(t)})}$$

$${\mathbf{\theta }}_{\lambda }^{(t+1)}\leftarrow$$

$${\mathbf{\xi }}^{(t+1)}\leftarrow$$

$${\mathbf{\theta }}_J^{(t+1)}$$

$${\mathbf{\theta }}_g^{(t+1)}$$

$${\mathbf{\theta }}_c^{(t+1)}$$

$${\mathbf{\theta }}_f^{(t+1)}$$

$${\mathbf{\theta }}_{\lambda }^{(t+1)}$$

$${\mathbf{\xi }}^{(t+1)}$$

$$\nabla_{\bm{\theta}_{f}}\mathbb{E}_{\mathbf{h}}\left[J(\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f}),\mathbf{h})\right]\\ =\mathbb{E}_{\mathbf{h}}\Big{[}\nabla_{\bm{\theta}_{f}}\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})\nabla_{{\mathbf{x}}}J(\mathbf{x},\mathbf{h})\big{|}_{\mathbf{x}=\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})}\Big{]}$$

$$P4:\underset{{\mathbf{\theta }}_{\pi }}{\max }$$

$$s.t.$$

$${\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }[\mathbf{c}(\mathbf{x},\mathbf{h})]⪯0$$

$${\nabla }_{{\mathbf{\theta }}_{\pi }}\mathcal{L}={\nabla }_{{\mathbf{\theta }}_{\pi }}{\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }\left[J-{\tilde{\mathbf{\lambda }}}^T\mathbf{g}-{\mathbf{\xi }}^T\mathbf{c}\right]$$

$$={\mathbb{E}}_{\mathbf{h}}\left[\sum _{\mathbf{x}}{\nabla }_{{\mathbf{\theta }}_{\pi }}\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi })(J-{\tilde{\mathbf{\lambda }}}^T\mathbf{g}-{\mathbf{\xi }}^T\mathbf{c})\right]$$

$$={\mathbb{E}}_{\mathbf{h}}\left[\sum _{\mathbf{x}}\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi })\frac{{\nabla }_{{\mathbf{\theta }}_{\pi }}\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi })}{\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi })}(J-{\tilde{\mathbf{\lambda }}}^T\mathbf{g}-{\mathbf{\xi }}^T\mathbf{c})\right]$$

$$={\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }\left[(J-{\tilde{\mathbf{\lambda }}}^T\mathbf{g}-{\mathbf{\xi }}^T\mathbf{c}){\nabla }_{{\mathbf{\theta }}_{\pi }}\log (\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi }))\right]$$

$${\nabla }_{{\mathbf{\theta }}_{\lambda }}\mathcal{L}=-{\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }\left[({\nabla }_{{\mathbf{\theta }}_{\lambda }}\tilde{\mathbf{\lambda }})\mathbf{g}\right]$$

$${\nabla }_{\mathbf{\xi }}\mathcal{L}=-{\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }[{\nabla }_{\mathbf{\xi }}\mathbf{c}]$$

$${\mathbf{\theta }}_{\pi }^{(t+1)}$$

$${\mathbf{\theta }}_{\lambda }^{(t+1)}$$

$${\mathbf{\xi }}^{(t+1)}$$

$$\begin{gathered} {\nabla }_{{\mathbf{\theta }}_f}{\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }\left[J(\mathbf{x},\mathbf{h})\right] \\ ={\mathbb{E}}_{\mathbf{h},\mathbf{x}\sim \pi }\left[J(\mathbf{x},\mathbf{h})\nabla \log (\pi (\mathbf{x}|\mathbf{h};{\mathbf{\theta }}_{\pi }))\right] \end{gathered}$$

$$\bm{\theta}_{\pi}^{(t+1)}\leftarrow\bm{\theta}_{\pi}^{(t)}+\delta_{\pi}\Big{[}(J-\mathbb{E}_{\mathbf{x}\sim\pi}[J])-\tilde{\bm{\lambda}}^{T}(\mathbf{g}-\mathbb{E}_{\mathbf{x}\sim\pi}[\mathbf{g}])\\ -\bm{\xi}^{T}(\mathbf{c}-\mathbb{E}_{\mathbf{x}\sim\pi}[\mathbf{c}])\Big{]}\nabla_{\bm{\theta}_{\pi}}\log(\pi(\mathbf{x}|\mathbf{h};\bm{\theta}_{\pi}^{(t)})$$

$${\mathbf{\theta }}_{\bar{J}}^{(t+1)}$$

$${\mathbf{\theta }}_{\bar{g}}^{(t+1)}$$

$${\mathbf{\theta }}_{\bar{c}}^{(t+1)}$$

$$P5:\underset{P(h)}{\max }$$

Full text

I Introduction

Various resource allocation and transceivers in wireless networks, such as power allocation, beamforming, and caching policy, can be designed by solving optimization problems with constraints, say imposed by the maximal transmit power, cache size, and the minimal data rate requirement [1, 2].

Depending on the applications, the objective function, constraints and the policy to be optimized may vary in different timescales. If they are in the same timescale, the problem is variable optimization, and the policy to be optimized is a vector with finite dimension, e.g., optimizing beamforming based on small scale channel gains to maximize the instantaneous data rate subject to instantaneous transmit power constraint. If they are in different timescales, the problem is functional optimization [3], i.e., the policy to be optimized is a function, which can be interpreted as a vector with infinite elements. A classical example of functional optimization is finding the instantaneous power allocation to maximize the ergodic capacity under the average power constraint, whose solution is the classical water-filling power allocation [4].

Variable optimizations have been well studied. Efficient tools, such as interior point method [5], have been developed to find the numerical optimal solutions of convex optimizations and various approximation methods have been widely used in non-convex optimizations. The optimal solutions of functional optimizations are, however, generally not in closed-form and inefficient to be obtained numerically. One way for numerical searching is the finite element method [6], which converts the functional optimization into a variable optimization by only optimizing the values of function on a finite sampled points. However, such method suffers from the curse of dimensionality. To overcome this shortage, an unsupervised learning framework was developed in [7], which parameterizes the functions by neural networks and trains the network parameters with stochastic gradient descent (SGD).

To apply numerical searching methods, the expressions of objective function and constraints, i.e., the model, should be known. For methods like interior point method and SGD, the gradients of objective function and constraints with respect to the optimization variables or functions are further required. However, in many scenarios, the expressions of the objective or constraints are unavailable, or too complex to derive their gradients. Finite difference method can be used to estimate the gradients according to the observations of the objective and constraints. However, such a method is inefficient when the objective function and the constraints are in high dimensions, and hence is not applicable for functional optimization problems, which are with infinite dimensions.

In this work, we propose a model-free framework to solve functional optimizations with instantaneous and average constraints without the supervision of optimal solution. We begin with a model-based framework for unsupervised learning where a neural network (called as policy network) is used for parameterizing the function to be optimized. We recast the original constrained problem in the dual domain where the Lagrangian is served as the objective function and another neural network (called as multiplier network) is introduced to parameterize the Lagrange multiplier associated with the instantaneous constraint. In the model-free framework, we resort to neural networks (called as value networks) to approximate the unavailable expressions of objective function or constraints so that their gradients can be obtained for training the policy and multiplier networks. Then, we reveal the connections of the proposed framework with reinforcement learning and show how to extend the proposed framework into stochastic policy optimization, which is applicable for both continuous and discrete policy optimizations. We study a simple power control problem to illustrate how to apply the framework and show the effectiveness of the model-free unsupervised learning.

II Unsupervised Learning for Optimizations

In this section, we introduce model-based and model-free unsupervised learning frameworks for solving general functional optimization problems. Since variable optimization can be treated as a special case of functional optimization, these frameworks are also applicable to variable optimization.

Let $\mathbf{h}\in\mathbb{R}^{n}$ denote a random vector reflecting environment status, e.g., channel gains. For each realization of $\mathbf{h}$, we aim to find a vector $\mathbf{x}\in\mathbb{R}^{m}$ to execute, e.g., transmit powers. Let function (called as a policy) $\mathbf{f}$: $\mathbb{R}^{n}\mapsto\mathbb{R}^{m}$ denote the mapping from $\mathbf{h}$ to $\mathbf{x}$, i.e., $\mathbf{x}=\mathbf{f}(\mathbf{h})$. The performance metric is a scalar function of $\mathbf{x}$ and $\mathbf{h}$ denoted by $J(\mathbf{x},\mathbf{h})$, e.g., the instantaneous data rate. The goal is to design a function $\mathbf{f}$ that maximizes the performance metric averaged over $\mathbf{h}$, i.e., $\mathbb{E}_{\mathbf{h}}[J(\mathbf{x},\mathbf{h})]$, subject to some constraints. This can be formulated as a general functional optimization problem as follows,

[TABLE]

where (1b) and (1c) denote the instantaneous and the average constraints, respectively, and the curled inequality symbol “$\preceq$” (or “$\succeq$”) denotes the element-wise inequality. It is noteworthy that optimization problems minimizing the objective function or having “$\succeq$” or “$=$” constraints (e.g., minimal data rate constraint) can be easily transformed into problem $\mathsf{P1}$.

In general, problem $\mathsf{P1}$ is hard to solve because it is a function $\mathbf{f}(\mathbf{h})$ that needs to optimize, which can be interpreted as vectors with infinite dimension when $\mathbf{h}$ is with infinity number of possible values.

II-A Model-Based Unsupervised Learning

To tackle with the constraints, we reconsider problem $\mathsf{P1}$ in its dual domain. The Lagrangian of $\mathsf{P1}$ can be written as [8]

[TABLE]

where $\bm{\lambda}(\mathbf{h})$ and $\bm{\xi}$ are the Lagrange multiplier associated with constraints (1b) and (1c), respectively. When strong duality condition holds [5], the original problem is equivalent to finding the saddle point of the Lagrangian as

[TABLE]

where $\bm{\lambda}({\mathbf{h}})$ is also a function that needs to be optimized. Thanks to the universal approximation theorem [9], we can introduce two neural networks to approximate function $\mathbf{f}$ as $\mathbf{f}(\mathbf{h})\approx\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})$ (called as policy network) and approximate multiplier as $\bm{\lambda}(\mathbf{h})\approx\tilde{\bm{\lambda}}(\mathbf{h};\bm{\theta}_{\lambda})$ (called as multiplier network), respectively, with arbitrary accuracy by finite-dimension parameter vectors $\bm{\theta}_{f}$ and $\bm{\theta}_{\lambda}$. Then, problem $\mathsf{P2}$ degenerates into the following variable optimization as

[TABLE]

To solve problem $\mathsf{P3}$, we can adopt the primal-dual stochastic gradient method [10] that iteratively updates the primal variable $\bm{\theta}_{f}$, and the dual variables $\bm{\theta}_{\lambda}$ and $\bm{\xi}$ along the ascent and descent directions of sample-averaged gradients, respectively. The gradients of the Lagrangian (4a) with respect to $\bm{\theta}_{f}$, $\bm{\theta}_{\lambda}$, and $\bm{\xi}$ can be derived as

[TABLE]

where $\nabla_{\mathbf{x}}y=[\frac{\partial y}{\partial x_{1}},\cdots,\frac{\partial y}{\partial x_{n}}]^{T}$ denotes the gradient, $\nabla_{\mathbf{x}}\mathbf{y}=[(\nabla_{\mathbf{x}}y_{1}),\cdots,(\nabla_{\mathbf{x}}y_{m})]$ denotes the transpose of Jacobian matrix and $(\cdot)^{T}$ is the transpose operation. Both $\nabla_{\bm{\theta}_{f}}\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})$ and $\nabla_{\bm{\theta}_{\lambda}}\tilde{\bm{\lambda}}(\mathbf{h};\bm{\theta}_{\lambda})$ can be computed via back propagation.

Let $\mathcal{B}$ denote a batch of realizations of $\mathbf{h}$. Then, the primal and dual variables are updated by

[TABLE]

where $\delta_{f}$, $\delta_{\lambda}$ and $\delta_{\xi}$ are learning rates, and $[\mathbf{x}]^{+}=[\max(x_{1},0),\cdots,\max(x_{2},0)]^{T}$. The operation $[\cdot]^{+}$ in (6) ensures constraint (4c). Constraints (4b) can be satisfied by properly chosen the activation function of the output layer of $\tilde{\bm{\lambda}}(\mathbf{h};\bm{\theta}_{\lambda})$, e.g., ReLU.

If the gradients $\nabla_{{\mathbf{x}}}J(\mathbf{x},\mathbf{h})$, $\nabla_{{\mathbf{x}}}\mathbf{g}(\mathbf{x},\mathbf{h})$ and $\nabla_{{\mathbf{x}}}\mathbf{c}(\mathbf{x},\mathbf{h})$ can be computed, an approximated optimal solution of problem $\mathsf{P3}$ can be obtained after the iterations in (6) converges.

II-B Model-Free Unsupervised Learning

For many problems in wireless networks, one or all of the objective and constraint functions in problem $\mathsf{P1}$ cannot be derived in closed-form, and we can only observe the values of these functions after executing $\mathbf{x}$ at a realization of $\mathbf{h}$ and then observe the values of $J(\mathbf{x},\mathbf{h})$, $\mathbf{g}(\mathbf{x},\mathbf{h})$, and $\mathbf{c}(\mathbf{x},\mathbf{h})$. For example, we can measure the data rate $J(\mathbf{x},\mathbf{h})$ after transmit with power $\mathbf{x}$ at channel state $\mathbf{h}$. For these scenarios, the gradients cannot be derived analytically. In what follows, we resort to model-free unsupervised learning that does not require the explicit expressions of these gradients.

Again according to the universal approximation theorem, we can approximate the objective function and constraints in problem $\mathsf{P1}$ by neural networks as $J(\mathbf{x},\mathbf{h})\approx\tilde{J}(\mathbf{x},\mathbf{h};\bm{\theta}_{J})$, $\mathbf{g}(\mathbf{x},\mathbf{h})\approx\tilde{\mathbf{g}}(\mathbf{x},\mathbf{h};\bm{\theta}_{g})$, and $\mathbf{c}(\mathbf{x},\mathbf{h})\approx\tilde{\mathbf{c}}(\mathbf{x},\mathbf{h};\bm{\theta}_{c})$. With the approximated objective function and constraints (called as value networks), the gradients can be then computed.

The values of $J(\mathbf{x},\mathbf{h})$, $\mathbf{g}(\mathbf{x},\mathbf{h})$, and $\mathbf{c}(\mathbf{x},\mathbf{h})$ can be measured and recorded in a system, which can be used as labels for training. Then, the neural networks $\tilde{J}(\mathbf{x},\mathbf{h};\bm{\theta}_{J})$, $\tilde{\mathbf{g}}(\mathbf{x},\mathbf{h};\bm{\theta}_{g})$, and $\tilde{\mathbf{c}}(\mathbf{x},\mathbf{h};\bm{\theta}_{c})$ can be trained by minimizing the $L_{2}$-norm loss function with stochastic gradient descent as

[TABLE]

where $\delta_{J}$, $\delta_{g}$ and $\delta_{c}$ are learning rates, $\mathcal{B}$ denotes a batch of tuples whose elements are the realizations of $\mathbf{h}$, the corresponding vector $\mathbf{x}$ conditioned on $\mathbf{h}$, and the values of $J(\mathbf{x},\mathbf{h})$, $\mathbf{g}(\mathbf{x},\mathbf{h})$ and $\mathbf{c}(\mathbf{x},\mathbf{h})$ measured after executing $\mathbf{x}$.

In the following, we denote $\mathbf{y}\triangleq\mathbf{y}(\cdot)$ for notational simplicity, e.g., $J\triangleq J(\mathbf{x},\mathbf{h})$ and $\tilde{J}\triangleq\tilde{J}(\mathbf{x},\mathbf{h};\bm{\theta}_{J})$. By substituting $\nabla_{\mathbf{x}}J\approx\nabla_{\mathbf{x}}\tilde{J}$, $\nabla_{\mathbf{x}}\mathbf{g}\approx\nabla_{\mathbf{x}}\tilde{\mathbf{g}}$, and $\nabla_{\mathbf{x}}\mathbf{c}\approx\nabla_{\mathbf{x}}\tilde{\mathbf{c}}$ into (6), we can obtain the update rule for $\bm{\theta}_{f}$, $\bm{\theta}_{\lambda}$, and $\bm{\xi}$ as

[TABLE]

Remark 1

When problem $\mathsf{P1}$ has no constraints, our model-free unsupervised learning framework degenerates into a special case of reinforcement learning, where the policy $\mathbf{f}(\mathbf{h})$ does not affect the distribution of state $\mathbf{h}$. For the unconstrained problem, the gradient of the Lagrangian with respect to policy parameter $\bm{\theta}_{f}$ in (5a) degenerates into

[TABLE]

which coincides with the deterministic policy gradient (DPG) theorem [11], where $J(\mathbf{x},\mathbf{h})$ is actually the action-value function (also known as Q-function or critic) and the policy network $\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})$ is the actor. By replacing $J$ in (9) with its approximation $\tilde{J}$, we can obtain the approximated policy gradient used for updating the actor in deep deterministic policy gradient (DDPG) algorithm [12].

Inspired by the great success of actor-critic approach in reinforcement learning, we can train the neural networks $\tilde{J}$, $\tilde{\mathbf{g}}$, $\tilde{\mathbf{c}}$, $\tilde{\mathbf{f}}$, and $\tilde{\bm{\lambda}}$ simultaneously via interactions with the environment. Each time after we observe the values of $J$, $\mathbf{g}$, and $\mathbf{c}$, we update parameters $\bm{\theta}_{J}$, $\bm{\theta}_{g}$, and $\bm{\theta}_{c}$ to obtain a better approximation of the Lagrangian. Meanwhile, we also update parameters $\bm{\theta}_{f}$, $\bm{\theta}_{\lambda}$, and $\bm{\xi}$ to improve the policy. Because $J$, $\mathbf{g}$, and $\mathbf{c}$ are functions of $\mathbf{x}$, to better approximate $J$, $\mathbf{g}$, $\mathbf{c}$ and their gradients at $\mathbf{x}$, it is necessary to obtain the values of $J$, $\mathbf{g}$, and $\mathbf{c}$ in the neighborhood of $\mathbf{x}$. To encourage such exploration, we add a noise term $\mathbf{n}^{(t)}$ that reduces over iterations to the output of policy network, i.e., $\mathbf{x}=\tilde{\mathbf{f}}+\mathbf{n}^{(t)}$. The detailed learning procedure is provided in Algorithm 1.

So far, we have implicitly assumed that the policy to be learned is continuous (i.e., $\mathbf{f}(\mathbf{h})$ is a continuous function of $\mathbf{h}$), and learn its parameterized form $\tilde{\mathbf{f}}(\mathbf{h};\bm{\theta}_{f})$ as a deterministic policy in both model-based and model free unsupervised learning frameworks. In some scenarios, we need to find a discrete policy, e.g., for user scheduling, where parameterizing a deterministic policy is not applicable because the output of neural network is continuous w.r.t the input. Although a discrete policy can be obtained by discretized a learned deterministic policy, the constraints may not be satisfied after the discretization.

Alternatively, we can parameterize a stochastic policy by neural network, which can be used to learn both continuous and discrete policies. Let $\pi(\mathbf{x}|\mathbf{h};\bm{\theta}_{\pi})$ denote the probability that we execute $\mathbf{x}$ conditioned on $\mathbf{h}$, and $\bm{\theta}_{\pi}$ is the network parameter. In this case, the parameterized form of problem $\mathsf{P1}$ becomes

[TABLE]

where $\mathbf{x}\sim\pi$ denotes that random variable $\mathbf{x}$ is sampled from distribution $\pi(\mathbf{x}|\mathbf{h};\bm{\theta}_{\pi})$, the objective function and average constraints are also averaged over $\mathbf{x}$. We can obtain the Lagrangian and use neural network $\tilde{\bm{\lambda}}(\mathbf{h};\bm{\theta}_{\lambda})$ to parameterize $\bm{\lambda}(\mathbf{h})$. Then, the gradient of Lagrangian with respect to $\bm{\theta}_{\pi}$ can be derived as

[TABLE]

The gradient of Lagrangian with respect to $\bm{\theta}_{\lambda}$ and $\bm{\xi}$ can be derived as

[TABLE]

Different from the deterministic policy case, the gradients $\nabla_{\mathbf{x}}J$, $\nabla_{\mathbf{x}}\mathbf{g}$, and $\nabla_{\mathbf{x}}\mathbf{c}$ are no longer necessary when we update $\bm{\theta}_{\pi}$, $\bm{\theta}_{\lambda}$, and $\bm{\xi}$ with stochastic gradient method. To compute a sample of the gradient in (11b)$\sim$(13), we only need to observe the value of $J$, $\mathbf{g}$, and $\mathbf{c}$ from the environment111When model is available, we can compute the values $J$, $\mathbf{g}$, and $\mathbf{c}$ from their expressions. when executing $\mathbf{x}$ at state $\mathbf{h}$. Therefore, $\bm{\theta}_{\pi}$, $\bm{\theta}_{\lambda}$, and $\bm{\xi}$ are updated by

[TABLE]

Remark 2

Although (11b) is derived assuming discrete distribution of $\mathbf{x}$, it can also be derived from a continuous distribution of $\mathbf{x}$. Therefore, (14) (and the following updating rules in (16)) are also applicable for learning a continuous policy.

Remark 3

When there are no constraints in problem $\mathsf{P4}$, (11b) reduces to

[TABLE]

which coincides with the policy gradient theorem [13] in reinforcement learning, and the update of $\bm{\theta}_{\pi}$ in (14) degenerates into the REINFORCE method [13].

The stochastic gradient update in (14a) may exhibit large variance [13] because the parameterized policy is stochastic and hence converge slowly. Inspired by the advantage actor-critic approach [14], we can subtract a term $\mathbb{E}_{\mathbf{x}\sim\pi}[J(\mathbf{x},\mathbf{h})-\tilde{\bm{\lambda}}^{T}\mathbf{g}(\mathbf{x},\mathbf{h})-\bm{\xi}^{T}\mathbf{c}(\mathbf{x},\mathbf{h})]$ into the parenthesis of (11a), which do not change the expectation of gradients but can reduce the variance. Then, the update for $\bm{\theta}_{\pi}$ becomes

[TABLE]

Again, the average terms can be approximated by neural networks as $\mathbb{E}_{\mathbf{x}\sim\pi}[J]\approx\bar{J}(\mathbf{h};\bm{\theta}_{\bar{J}})$, $\mathbb{E}_{\mathbf{x}\sim\pi}[\mathbf{g}]\approx\bar{\mathbf{g}}(\mathbf{h};\bm{\theta}_{\bar{g}})$ and $\mathbb{E}_{\mathbf{x}\sim\pi}[\mathbf{c}]\approx\bar{\mathbf{c}}(\mathbf{h};\bm{\theta}_{\bar{c}})$, which are updated by minimizing the $L_{2}$-norm loss with stochastic gradient descent as

[TABLE]

The detailed learning procedure is provided in Algorithm 2.

III Case Study: Power Control Problem

In this section, we illustrate how to apply the model-based and model-free unsupervised learning frameworks for solving optimization problems. For easy understanding, we consider a simple power control problem in point to point communications. To provide a baseline, we first derive the analytical solution of the problem. Then, we show how to employ the frameworks when the expression of the objective function is known and unknown.

In what follows we optimize the instantaneous transmit power to minimize the ergodic capacity under the constraints of average transmit power and maximum transmit power,

[TABLE]

where $h$ is the small-scale channel gain, $P(h)$ is the transmit power adapted to $h$, $R(P(h),h)$ is the channel capacity, $\bar{P}>0$ is the maximum average transmit power, and $P_{\max}>\bar{P}$ is the maximum instantaneous transmit power.

III-A Analytical Solution

When the channel coding is sufficient long and the noise is Gaussian distributed, the channel capacity can be expressed as the Shannon’s formula, i.e., $R\left(P(h),h\right)\!=\!\log_{2}\!{(1\!+\!\frac{hP(h)}{N})}$, where $N\!>\!0$ is the power of noise unified by the large-scale channel gain. Then the Karush-Kuhn-Tucker (KKT) conditions of problem (18) can be derived as [8],

[TABLE]

As proved in the Appendix, the solution of the problem is,

[TABLE]

where $\xi^{*}$ satisfies $\mathbb{E}_{h}\left[P^{*}(h)\right]=\bar{P}$ and can be computed via bisection searching with known distribution of $h$. The solution in (23) differs from the water-filling structure [4] due to the additional constraint imposed by $P_{\max}$.

III-B Model-Based Unsupervised Learning Method

Problem $\mathsf{P5}$ may not have closed-form solution, say when the finite block-length channel coding is used such that $R\left(P(h),h\right)$ is with complex expression. In the sequel, we illustrate how to use model-based unsupervised learning to solve the problem.

The function to be optimized is approximated by a policy network $\tilde{P}(h;\bm{\theta}_{P})$. The constraints in (18b) can be satisfied by setting the active function of the output layer in $\tilde{P}(h;\bm{\theta}_{P})$ as Sigmoid, and multiplying the final output by $P_{\max}$. However, to validate the effectiveness of the multiplier network in handling the instantaneous constraints in functional optimization problems, we use ReLU as the active function of the output layer to only ensure $\tilde{P}(h;\bm{\theta}_{P})\geq 0$, and introduce the multiplier network $\tilde{\lambda}(h;\bm{\theta}_{\lambda})$ to ensure the constraint $\tilde{P}(h;\bm{\theta}_{P})\leq P_{\max}$ with primal-dual stochastic gradient method given by (8). Then, the power control policy and the Lagrange multipliers can be updated by

[TABLE]

where $\delta_{P}$, $\delta_{\lambda}$, and $\delta_{\xi}$ are the learning rates, and $\mathcal{B}$ denotes a batch of training samples.

III-C Model-Free Unsupervised Learning Method

When the channel coding is short or the noise is not Gaussian distributed, the Shannon’s formula is not applicable and the expression of $R\left(P(h),h\right)$ is hard to obtain. In the following, we illustrate how to use the proposed model-free unsupervised learning to solve problem $\mathsf{P5}$.

The objective function is approximated by introducing the value network $\tilde{R}(P,h;\bm{\theta}_{R})$, which is then used to compute the approximated gradient $\nabla_{P}\tilde{R}\approx\nabla_{P}R$ for updating the policy network parameter $\bm{\theta}_{P}$ in (24a). The updates for $\bm{\theta}_{\lambda}$ and $\xi$ are the same as in (24b) and (24c) since the expressions of constraints are known. After observing the actual data rate acheived by transmiting with power $P$ at channel state $h$, the value network $\tilde{R}(P,h;\bm{\theta}_{R})$ is trained based on the observed value of $R(P,h)$ according to (7) as

[TABLE]

where $\delta_{R}$ denotes the learning rate.

IV Numerical and Simulation Results

In this section, we validate the proposed mode-free unsupervised learning frameworks by considering problem $\mathsf{P5}$, where in simulation the channel coding is assumed long and the noise is assumed Gaussian.

The simulation setup is as follows. The maximal instantaneous and average transmit powers are $P_{\max}=40$ W and $\bar{P}=30$ W, respectively. The distance between the transmitter and the receiver is $d=500$ m. The noise power spectral density is $-174$ dBm/Hz and the bandwidth is $20$ MHz. We consider Rayleigh fading channels and the path loss is modeled by $35.3+37.6\log_{10}(d)$ in dB.

The hyper-parameters used for model-based and model-free frameworks are as follows. Both $\tilde{\lambda}$ and $\tilde{P}$ have three fully-connected hidden layers with $50$, $40$, and $30$ nodes, respectively. $\tilde{R}$ has two hidden layers with $200$ and $150$ nodes, respectively. All the hidden layers and the output layers of $\tilde{\lambda}$ and $\tilde{P}$ use ReLU as the activation function. The output layer of $\tilde{R}$ has no activation function. We use Adam [15] for training all the neural networks with learning rate $\delta_{P}=\delta_{\lambda}=10^{-3}$ for $\tilde{P}$ and $\tilde{\lambda}$, and $\delta_{R}=5\times 10^{-3}$ for $\tilde{R}$. The batch size is $|\mathcal{B}|=32$. $\tilde{P}$ is initialized as $10$. Both $\tilde{\lambda}$ and $\xi$ are initialized as [math]. The noise term for exploration in model-free learning is set as $n^{(t)}=\epsilon^{(t)}\mathcal{N}^{(t)}$ where $\mathcal{N}^{(t)}$ denotes Gaussian noise with zero mean and unit variance. The value of $\epsilon^{(t)}$ is set as $10$ for the first $5\times 10^{3}$ iterations and then decreases linearly to zero for the next $1.5\times 10^{4}$ iterations. All the simulation results are averaged over $50$ rounds of learning.

In Fig. 1, we compare the convergence of model-based and model-free unsupervised learning. Both model-based and model-free learning can converge to the average rate achieved by the optimal solution $P^{*}(h)$ numerically computed with (23) (with legend “Optimal”). The average rate achieved by model-free learning can be even higher than the optimal solution at the beginning due to violation of constraints. We show the violations of instantaneous and average constraints in Fig. 1(b) and Fig. 1(c), respectively. Since model-free learning needs the exploration to learn the expression of the objective function, the violations of constraints are more severe than model-based method at the beginning of learning due to insufficient training samples. With the increase of iterations, both model-based and model-free learning can satisfy all the constraints after convergence. Moreover, the number of iterations for converging to the optimal solution are close for model-based and model-free learning. This demonstrates the efficiency of proposed model-free unsupervised learning framework where the policy, multiplier, and value networks are trained simultaneously.

In Fig. 2, we compare the behavior of the policies learned by model-based and model-free frameworks with the optimal solution. We can see that the learned policies behave almost the same with optimal policy.

V Conclusions

In this paper, we proposed an framework to solve optimization problems with constraints by model-free unsupervised learning, and revealed the connections with reinforcement learning. We illustrated how to apply the proposed framework by a power control problem. Numerical and simulation results validated our framework and showed that model-free unsupervised learning can converge to the optimal policy with similar speed as model-based unsupervised learning.

To find the solution from the KKT conditions in (19), $P^{*}(h)$, $\lambda_{1}^{*}(h)$, $\lambda_{2}^{*}(h)$, and $\xi^{*}$, we first prove $\xi^{*}>0$.

Assume $\xi^{*}=0$. Since $\frac{1}{N/h+P^{*}(h)}>0$ and $\lambda_{1}^{*}(h)\geq 0$, we have $\lambda_{2}^{*}(h)>0$ according to (19a). Then, $P^{*}(h)=P_{\max}$ can be derived from (19d). In this case, $\mathbb{E}_{h}\left[P(h)\right]=P_{\max}>\bar{P}$, which violate the constraint in (18a). Therefore, $\xi^{*}>0$. From (19b), we further have,

[TABLE]

When $h<\xi^{*}N$, we have $\frac{1}{N/h+P^{*}(h)}-\xi^{*}<0$. In this case, $\lambda_{1}^{*}(h)>0$ according to (19a) and $P^{*}(h)=0$ according to (19c). When $h=\xi^{*}N$, $\frac{1}{N/h+P^{*}(h)}-\xi^{*}<0$ if P^{*}(h)>$$~{}0, which on the contrary results in $P^{*}(h)=0$. Therefore, $P^{*}(\xi^{*}N)=0$.

When $h>\frac{N}{1/\xi^{*}-P_{\max}}$, we have $\frac{1}{N/h+P^{*}(h)}-\xi^{*}>0$. In this case, $\lambda_{2}^{*}(h)>0$ according to (19a) and $P^{*}(h)=P_{\max}$ according to (19d). When $h=\frac{N}{1/\xi^{*}-P_{\max}}$, $\frac{1}{N/h+P^{*}(h)}-\xi^{*}>0$ if $P^{*}(h)<P_{\max}$, which results in $P^{*}(h)=P_{\max}$, contradicting with $P^{*}(h)<P_{\max}$. Therefore, $P^{*}(\frac{N}{1/\xi^{*}-P_{\max}})=P_{\max}$.

When $\xi^{*}N<h<\frac{N}{1/\xi^{*}-P_{\max}}$, we have $\frac{1}{N/h+P_{\max}}<\xi^{*}<\frac{1}{N/h+0}$. In this case, if $P^{*}(h)=0$, then $\lambda_{2}^{*}(h)>0$ according to (19a), which results in $P^{*}(h)=P_{\max}$, contradicting with $P^{*}(h)=0$. Similarly, if $P^{*}(h)=P_{\max}$, then $\lambda_{1}^{*}(h)>0$ according to (19a), which results in $P^{*}(h)=0$, contradicting with $P^{*}(h)=P_{\max}$. Therefore, we have $0<P^{*}(h)<P_{\max}$ and $\lambda_{1}^{*}(h),\lambda_{2}^{*}(h)=0$. According to (19a) we further have $P^{*}(h)=1/\xi^{*}-N/h$.

Finally, according to the solution of $P^{*}(h)$, $\xi^{*}$ can be solved from (26).