Computing the channel capacity of a communication system affected by uncertain transition probabilities

TL;DR

This paper develops a first-order optimization algorithm to compute the capacity of uncertain discrete memoryless channels efficiently, especially for large-scale problems, by transforming the max-min problem into a convex optimization problem.

Contribution

It introduces an $ ext{O}(1/T)$ first-order method for solving the robust channel capacity problem under uncertainty, improving scalability over interior point methods.

Findings

Algorithm effectively computes channel capacity under uncertainty.

Applicable to large-scale problems where traditional methods are impractical.

Transforms max-min problem into a convex optimization problem.

Abstract

We study the problem of computing the capacity of a discrete memoryless channel under uncertainty affecting the channel law matrix, and possibly with a constraint on the average cost of the input distribution. The problem has been formulated in the literature as a max-min problem. We use the robust optimization methodology to convert the max-min problem to a standard convex optimization problem. For small-sized problems, and for many types of uncertainty, such a problem can be solved in principle using interior point methods (IPM). However, for large-scale problems, IPM are not practical. Here, we suggest an $\mathcal{O}(1/T)$ first-order algorithm based on Nemirovski (2004) which is applied directly to the max-min problem.

Equations

$$\underset{p\in {\Delta }_N}{\sup }I(p,Q):=\underset{p\in {\Delta }_m}{\sup }\sum _{n=1}^N\sum _{m=1}^M{p}_n{Q}_{nm}\log \frac{Q_{nm}}{\sum _{l=1}^N{p}_l{Q}_{lm}},$$

$$\underset{p\in {\Delta }_m}{\sup }\underset{Q\in \mathcal{Q}}{\inf }I(p,Q),$$

$$\underset{p,\lambda ,V}{\max }$$

$$\sum _{n=1}^N{p}_n\exp \left(\frac{{\lambda }_n-V_{nm}}{p_n}\right)\le 1,\forall m$$

$$p\ge 0$$

$$\sum _{n=1}^N{p}_n=1,$$

$$f(Q,p)=\sum _{n=1}^N\sum _{m=1}^M{p}_n{Q}_{nm}\log \frac{Q_{nm}}{\sum _{l=1}^N{p}_l{Q}_{lm}}.$$

$$\underset{Q\in \mathcal{Q}}{\inf }f(Q,p)$$

$$=\underset{Q}{\inf }\left\{\delta \left(Q|\mathcal{Q}\right)-(-f(Q,p))\right\}$$

$$=\underset{V}{\sup }\left\{(-f{)}_{\ast }(V,p)-{\delta }^{\ast }\left(V|\mathcal{Q}\right)\right\},$$

$$(-f{)}_{\ast }(V,p)=\underset{Q\in {\Delta }_{N\times M}}{\inf }\sum _{n=1}^N\sum _{m=1}^M\left(p_n{Q}_{nm}\log \frac{Q_{nm}}{\sum _{l=1}^N{p}_l{Q}_{lm}}+V_{nm}{Q}_{nm}\right)$$

$$x_{nm}=p_n{Q}_{nm},\forall n,m,q_m=\sum _{n=1}^N{p}_n{Q}_{nm}$$

$$\underset{Q\in {\Delta }_{N,M}}{\inf }\sum _{n=1}^N\sum _{m=1}^M\left(p_n{Q}_{nm}\log \frac{Q_{nm}}{\sum _{l=1}^N{p}_l{Q}_{lm}}+V_{nm}{Q}_{nm}\right)$$

$$\underset{x_{nm},q_m}{\inf }$$

$$\sum _{m=1}^M{x}_{nm}=p_n\forall n$$

$$q\in {\Delta }_M$$

$$x_{nm}\ge 0\forall n,m.$$

$$\underset{q\in {\Delta }_M,x_{nm}\ge 0}{\inf }\sum _{n=1}^N\sum _{m=1}^M\left(x_{nm}\log \frac{x_{nm}}{q_m}+V_{nm}{Q}_{nm}\right)-\sum _{n=1}^N{u}_n\left(\sum _m{x}_{nm}-p_n\right)$$

$$=\sum _{n=1}^N{u}_n{p}_n+\underset{q\in {\Delta }_n}{\inf }\underset{x_{nm}\ge 0}{\inf }\sum _{n=1}^N\sum _{m=1}^M{q}_m\left(\frac{x_{nm}}{q_m}\log \frac{x_{nm}}{q_m}-\frac{x_{nm}}{q_m}\left(u_n-\frac{V_{nm}}{p_n}\right)\right)$$

$$=\sum _{n=1}^N{u}_n{p}_n+\underset{q\in {\Delta }_n}{\inf }\sum _{n=1}^N\sum _{m=1}^M{q}_m\exp \left(u_n-\frac{V_{nm}}{p_n}-1\right)$$

$$=\sum _{n=1}^N{u}_n{p}_n-\underset{m}{\max }\sum _{n=1}^N\exp \left(u_n-\frac{V_{nm}}{p_n}-1\right)$$

$$(-f{)}_{\ast }(V,p)=\underset{u_n,\mu }{\max }$$

$$\mu \ge \sum _{n=1}^N\exp \left(u_n-\frac{V_{nm}}{p_n}-1\right)$$

$$(-f{)}_{\ast }(V,p)=\underset{\lambda ,\mu }{\max }$$

$$\mu \ge \sum _{n=1}^N{p}_n\exp \left(\frac{{\lambda }_n-V_{nm}}{p_n}-1+\mu \right)$$

$$A\ge A^{\prime }>0\Rightarrow \left(\mu \ge A\exp (\mu )\Rightarrow \mu \ge A^{\prime }\exp (\mu )\right).$$

$$\displaystyle\mu\geq\sum\limits_{n=1}^{N}p_{n}\exp\left(\frac{\lambda_{n}-V_{nm}}{p_{n}}-1+\mu\right)\$$

$$(-f{)}_{\ast }(V,p)=\underset{\lambda }{\max }$$

$$\sum _{n=1}^N{p}_n\exp \left(\frac{{\lambda }_n-V_{nm}}{p_n}\right)\le 1$$

$$\displaystyle\min_{v,Q}\$$

$$Q\in \mathcal{Q}.$$

$$\displaystyle\min(\ref{eq.cc.rc.dual})\leq\min_{Q\in\mathcal{Q}}\$$

$$Q=\left[\begin{array}[]{ccc}\frac{1}{3}&\frac{1}{6}&\frac{1}{2}\\ \frac{1}{3}&\frac{1}{2}&\frac{1}{6}\end{array}\right]$$

$$\mathcal{Q}=\text{conv}\left(\left[\begin{array}[]{ccc}\frac{1}{3}&\frac{1}{6}&\frac{1}{2}\\ \frac{1}{3}&\frac{1}{2}&\frac{1}{6}\end{array}\right],\left[\begin{array}[]{ccc}\frac{1}{3}&\frac{1}{2}&\frac{1}{6}\\ \frac{1}{3}&\frac{1}{6}&\frac{1}{2}\end{array}\right]\right).$$

$$\min (\text{ref}eq.cc.rc.dual)=\underset{Q\in \mathcal{Q}}{\min }\log N+\sum _{m=1}^M{Q}_{1m}\log \frac{Q_{1m}}{\sum _l{Q}_{lm}}.$$

$$Q=\left[\begin{array}[]{cc}1-\beta&\beta\\ \beta&1-\beta\end{array}\right].$$

$$\log 2+{\beta }^{\ast }\log {\beta }^{\ast }+(1-{\beta }^{\ast })\log (1-{\beta }^{\ast })\text{ where }{\beta }^{\ast }=\min \{1/2,\overline{\beta }\}.$$

$$\mathcal{Q}=\left\{r\in {\mathbb{R}}_{+}^M:1^{\top }r=1,\sum _{m=1}^M{r}_m\log \left(\frac{r_m}{q_m}\right)\le \rho \right\}.$$

Full text

1 Introduction

Channel capacity is a fundamental notion in the field of information theory that started with the seminal paper of [12]. For a given channel, it provides an upper bound on the rate at which information can be reliably transmitted. In this paper we consider the problem of computing the capacity of a discrete memoryless channel (DMC). A DMC consists of an input alphabet of $N$ symbols, each used with probability $p_{n}$ determined by the user, and an output alphabet $Y$ of $M$ symbols. Given the $n$-th input symbol, the $m$-th output symbol occurs with a conditional probability $Q_{nm}$, where the matrix $Q\in\mathbb{R}^{N\times M}$ is known as the channel law matrix. The channel capacity of a DMC is given by:

[TABLE]

where the maximized term is known as the average mutual information (AMIM). The AMIM is a convex function of the input distribution $p$ and hence, at least in theory, convex optimization algorithms can be used to solve (CC). However, the performance of such algorithms can be poor and the need for other approaches emerged. One of the first algorithms is the well-known Arimoto-Blahut algorithm (see [1; 6]). Later, more efficient algorithms have been developed to solve the capacity problem and some of its special cases. A comprehensive survey of algorithms for solving (CC) is given in [13]. This paper also suggests a first-order algorithm based on [11] for problem (CC) with an additional linear constraint.

In reality, channels correspond to physical devices affected by inaccuracies and/or whose parameters are measured only up to a certain accuracy. Therefore, the channel at hand may differ from its assumed version. Already in the 1950’s researchers were considering the possibility that the channel is not known fully. This case, known as compound channel case, has a rich literature, see e.g. [5; 8; 14]. A comprehensive survey by [9] summarizes these results. In this paper, the uncertain channel capacity problem is represented as:

[TABLE]

where the channel law matrix $Q$ is only known to belong to a set $\mathcal{Q}$. In today’s language (R-CC) could be called a robust optimization problem [3; 4] where $\mathcal{Q}$ is the so-called uncertainty set. To the best of our knowledge, there are no papers in the literature on algorithms for solving (R-CC). The main purpose of our paper is to fill this gap by designing an efficient algorithm for solving large scale instances. Contrary to [13], we cannot use the algorithm of [11] since the coupling term in (R-CC) is not bilinear. Instead, we propose a first-order algorithm based on [10]. The performance of this algorithm relies on two types of operation that need to be executed in each iteration. The first of them is the computation of the gradient of the AMIM with respect to $p$ and $Q$. Secondly, at each iteration the algorithm calls an oracle consisting of a proximal minimization. We show that for our problem, an analytic expression for the minimum is obtained, or the proximal minimization reduces to solving a single-variable convex problem.

The contribution of our paper is as follows:

• •

we show that the robust counterpart (RC) of the min-max problem (R-CC) reduces to a standard deterministic convex optimization problem. Moreover, we derive a dual form of RC. For the binary symmetric channel, this reduces to an explicit formula and for a weakly symmetric channel with Kullback-Leibler uncertainty it reduces to a single-variable problem;

• •

we adapt the algorithm of [10] to (R-CC) and derive simple conditions under which the $\mathcal{O}(1/T)$ convergence can be proved;

• •

similar to Sutter et al. [13], we look at an extended version of problem (R-CC) which includes an average cost constraint, and we show how to extend the algorithm to handle this case;

• •

by numerical experiments, we illustrate the effect of uncertainty on the channel capacity.

The organization of the paper is as follows. In Section 2 we derive the convex formulation of the uncertain channel capacity problem and its dual. In Section 3 we adapt the algorithm of [10] to solve the uncertain channel capacity problem for large-scale instances. Section 4 includes three numerical experiments illustrating the impact of uncertainty on channel capacity.

Notation. We denote by $\Delta_{N}$ the $N$-dimensional simplex and by $\Delta_{N\times M}$ is a set of matrices where each row belongs to $\Delta_{M}$. $C^{1,1}(A)$ shall denote the space of differentiable functions with a Lipschitz continuous gradient over the set $A$. By $\partial g(\lambda)$ we denote the subdifferential of function $g(\cdot)$ at point $\lambda$. For a norm $\|\cdot\|$ we denote by $\|\cdot\|_{*}$ its dual norm.

2 The uncertain channel capacity problem

2.1 Deriving the robust counterpart (RC)

The RC is derived by dualizing the inner problem in the min-max (R-CC) (Proposition 1). This results in a standard maximization of a concave function over a convex set.

Proposition 1

Let the uncertainty set $\mathcal{Q}\subset\mathbb{R}^{N\times M}$ be a compact convex set. Then, the worst-case capacity of a channel is given by the optimal value to the following problem:

[TABLE]

*where $\delta^{\ast}(V|\mathcal{Q})$ is the support function of the set $\mathcal{Q}$ evaluated at $V\in\mathbb{R}^{N\times M}$. *

Proof.

We begin by deriving the dual of the inner problem in (R-CC). Let:

[TABLE]

Using the results of [4], relying on Fenchel duality, we have:

[TABLE]

where $(-f)_{\ast}(V,p)$ is the concave conjugate of the function $-f$ with respect to the first variable and $\delta^{\ast}\left(V|\mathcal{Q}\right)$ is the support function of the set $\mathcal{Q}$, evaluated at $V\in\mathbb{R}^{N\times M}$. Therefore, we need to find:

[TABLE]

So, let us define new variables:

[TABLE]

in terms of which

[TABLE]

is equivalent to

[TABLE]

Next, we use Lagrangian duality to compute the dual of the latter problem :

[TABLE]

Introducing an auxiliary variable $\mu$ for the $\max$ term and using strong duality, we obtain:

[TABLE]

We introduce auxiliary variable $\lambda_{n}$ such that $u_{n}=\log p_{n}+\lambda_{n}/p_{n}+\mu$ to obtain

[TABLE]

Each of the constraints has a form $\mu\geq A\exp(\mu)$ where $A>0$. The property of such a constraint is:

[TABLE]

Therefore, if there exists a $\mu$ satisfying one of the constraints for which the expression $\sum_{n=1}^{N}p_{n}\exp\left((\lambda_{n}-V_{nm})/p_{n}-1\right)$ is smallest, this $\mu$ satisfies all of the constraints. Therefore:

[TABLE]

and one can eliminate $\mu_{m}$ from each inequality by moving the terms to the left-hand side and maximizing w.r.t. $\mu_{m}$. In the end, we obtain:

[TABLE]

This expression is inserted back into (1), which in turn inserted into (R-CC) results in the final form (P). ∎

Problem (P) is convex – the objective is a concave function of the decision variables and the first constraint involves the perspectives of the (convex) exponential function. Thus if the support function $\delta^{\ast}(\cdot|\mathcal{Q})$ has a tractable form (see e.g. [4] for a review of computationally tractable uncertainty sets), it can be solved using convex optimization methods, e.g., the IPM. In the next section, we derive the problem dual to (P) which allows us to obtain upper bounds on the robust channel capacity.

Remark 1

In [13], the deterministic channel capacity problem includes a linear average cost constraint. It models the case where use of the $n$-th symbol of the input alphabet incurs a cost $a_{n}$ and the aim is to keep the average cost below a threshold $b$. This gives a constraint $a^{\top}p\leq b$, where $a\in\mathbb{R}^{N}_{+}$ and $b\in\mathbb{R}_{+}$. In our robust setting, such a constraint is also easily included as it is sufficient to append it to problem (P).

2.2 Dual problem

The following proposition gives the convex optimization problem dual to (P).

Proposition 2

A dual of problem (P) for which strong duality holds is given by:

[TABLE]

Proof.

See Appendix A.1. ∎

The following corollary gives an upper bound on the robust channel capacity by inserting the feasible solution $v_{m}=\log\sum_{l}Q_{lm}$.

Corollary 1

The following is an upper bound on the robust channel capacity (R-CC):

[TABLE]

As it turns out, the upper bound of Corollary 1 is tight for the class of weakly symmetric channels.

Definition 1

A channel is weakly symmetric if each row contains the same set of values as any other row, with possible permutations, and the column sums are equal.

For example, the following channel is weakly symmetric:

[TABLE]

and the following uncertainty set involves only weakly symmetric channels:

[TABLE]

For weakly symmetric channels, it is sufficient to specify an uncertainty set for $Q$ by specifying the uncertainty set for a single row (with permuted values of the same entries in other rows) with the condition that the sums of the column entries stays equal.

Proposition 3

For weakly symmetric channels the bound (3) is tight:

[TABLE]

Proof.

See Appendix A.2. ∎

2.3 Special cases

As an illustration of the results, we consider two simple cases: (i) the well-known binary symmetric channel, and (ii) the weakly symmetric channel under Kullback-Leibler uncertainty about the transition probabilities in each row of $Q$.

2.3.1 Binary symmetric channel

A binary symmetric channel is one where $m=n=2$ and

[TABLE]

Clearly, this channel is characterized by a single parameter $\beta$ and hence, the only possible compact convex uncertainty set is an interval $\beta\in[\underline{\beta},\overline{\beta}]$. Then, the following result holds:

Proposition 4

Assume w.l.o.g. that $\underline{\beta}\leq 1/2$. Then, the robust capacity of a binary symmetric channel is given by

[TABLE]

Proof.

See Appendix A.3. ∎

2.3.2 Weakly symmetric channels under Kullback-Leibler (KL) uncertainty

Weak symmetry allows us to focus on a single row, so let us define $r_{m}=Q_{1m}$ and $\sum_{l=1}^{N}Q_{lm}=N/M$ for all $m$. Under KL uncertainty, the uncertainty set for the first row is given as:

[TABLE]

In this setting, the dual problem (D) reduces to the following one:

[TABLE]

Omitting the first term $\log M$, we write the Lagrangian and minimize w.r.t. $r_{m}$ to obtain the dual function:

[TABLE]

where $\lambda\geq 0$. To solve the dual problem $\max_{\mu,\lambda\geq 0}g(\mu,\lambda)$ we maximize first w.r.t. $\mu$ to obtain the optimality condition:

[TABLE]

which, after inserting it back, gives the following maximization problem over a single variable $\lambda\geq 0$:

[TABLE]

3 A proximal algorithm for solving the robust channel capacity problem (R-CC)

3.1 Introduction

In principle, the robust channel capacity problem of Proposition 1 can be solved using the interior point methods for a wide range of convex and compact uncertainty sets $\mathcal{Q}$. However, the numerical performance of these methods is poor already on medium-sized instances which is in line with the experience already gained in the deterministic case [13]. This is particularly due to the presence of the variable matrix $V\in\mathbb{R}^{N\times M}$, which greatly increases the computational effort since the IPMs in each iteration require solving a linear system of size corresponding to the number of variables. This situation calls for using first-order methods.

It turns out that in a suitable uncertainty setting, the original max-min problem (R-CC) can be solved using the prox-algorithm of [10]. The algorithm has a convergence rate of $\mathcal{O}(1/T)$ and its performance relies on two types of operation that need to be executed in each iteration: (i) computing the gradient of the AMIM with respect to $p$ and $Q$; (ii) calling an oracle which performs a proximal minimization step. For the max-min problem (R-CC) the partial derivatives of the AMIM with respect to $p$ and $Q$ are easily computed. As for the proximal minimization, an analytic expression for the minimum is obtained, or the proximal minimization reduces to solving a single-variable convex problem.

In Appendix B we state a self-contained description of the algorithm of [10]. Here, we first present our modelling setup and show that it satisfies the crucial assumptions for proving the convergence of the algorithm. In Section 3.2 we outline the algorithm and state its convergence. In Section 3.3 we extend the algorithm to the case with the average cost constraint added.

We propose a rather general modeling setup where the uncertainty set for the channel law matrix $Q$ is given as:

[TABLE]

where $\mathcal{B}\subset\mathbb{R}^{S}$ is a ‘simple’ compact convex set of perturbations $\xi$. By ‘simple’ it is meant that it is easy to perform the proximal minimization step. Examples of such simple $\mathcal{B}$ are: $\infty$-norm ball, 2-norm ball, and simplex. For the first two cases, we add to the description of the $\mathcal{Q}$ constraints $Q^{s}\mathbf{1}=\mathbf{0}$ for all $s=1,\ldots,S$, necessary for the row sums of $Q(\xi)$ to stay equal to $1$ for all $\xi\in\mathcal{B}$.

In this setup, the problem to solve is:

[TABLE]

where

[TABLE]

and where the fact that the max-min $=$ min-max holds due to the Sion-Kakutani minimax theorem. We set $z=(\xi,p)$ to shorten the notation and define the gradient mapping

[TABLE]

The three crucial assumptions for the algorithm’s convergence are: (i) compactness of the domain, (ii) Lipschitz continuity of $\Phi(z)$, (iii) existence of strongly convex distance-generating functions that it is easy to minimize them over the domain.

The compactness assumption (i) is satisfied due to the compactness of $\mathcal{B}$ and $\Delta_{N}$. Assumption (ii) can be easily met due to the following result.

Proposition 5

If for some $\tau>0$ it holds that $Q_{nm}>\tau$ for all $Q\in\mathcal{Q}$, then the mapping:

[TABLE]

is Lipschitz continuous on $\mathcal{Q}\times\Delta_{N}$. $\square$

Proof.

See Appendix A.4. ∎

Remark 2

Note the extra assumption $\tau>0$. A similar assumption is made in the deterministic channel case of [13], in order to force the set of dual solutions of a certain problem to be compact, on which the convergence of their algorithm rests.

The third assumption (iii) is the existence of a distance-generating function $\omega:\mathbb{R}^{n}\times\mathbb{R}_{+}^{m}\rightarrow\mathbb{R}$ such that the computation of the proximal operator:

[TABLE]

is simple. In line with [10], we use

[TABLE]

where $\omega_{1}(\cdot)$ and $\omega_{2}(\cdot)$ are strongly convex distance-generating functions over $\mathcal{B}$ and $\Delta_{N}$ and $\gamma_{1}$, $\gamma_{2}$ are suitably chosen positive constants. Possible choices for $\omega_{1}(\cdot)$ and $\omega_{2}(\cdot)$ are:

• •

$\omega_{1}(\cdot)$:

• for $\mathcal{B}$ – $\infty$-norm ball or $2$-norm ball:

[TABLE]

• for $\mathcal{B}$ – simplex:

[TABLE]

where $\delta>0$ is an arbitrarily positive number.

• •

$\omega_{2}(\cdot)$:

[TABLE]

With such a choice, the computation of the prox-operator is equivalent to:

[TABLE]

and can be done easily. Indeed, one can minimize (7) over $p$ and $\xi$ separately to obtain:

[TABLE]

where $\mu$ is the solution to the single-variable equation:

[TABLE]

For minimization over $\xi$ we have the following three cases corresponding to the different choices for $\omega_{1}(\cdot)$:

• •

$\mathcal{B}$ – $\infty$-norm ball:

[TABLE]

• •

$\mathcal{B}$ – 2-norm ball:

[TABLE]

• •

$\mathcal{B}$ – simplex is analogous to (15).

3.2 The algorithm and its convergence

Now, we present the scheme of the algorithm. Each general step is illustrated with the specific case where $\mathcal{B}$ is the $\infty$-norm ball, $\omega_{1}(\xi)=\xi^{\top}\xi/2$ and $\omega_{2}(p)=\sum\limits_{n=1}^{N}(p_{n}+\delta/N)\log(p_{n}+\delta/N)$.

Initialize with a point $(\xi^{0},p^{0})=z^{0}\in Z=\mathcal{B}\times\Delta_{N}$. 2. 2.

Given $z^{t-1}$, check if

[TABLE]

In our example, this reduces to solving the problem:

[TABLE]

and checking if $w=z^{t-1}$.

If equality holds, the algorithm stops and $z^{t-1}$ is the saddle point in (5). Otherwise, go to Step 3. 3. 3.

Choose a $\gamma^{t}>0$, set $w^{t,0}:=z^{t-1}$ and do the inner iteration (over $k=1,2,\ldots$):

[TABLE]

until for the first time the following inequality holds:

[TABLE]

In our example, this means solving the problem:

[TABLE]

until it holds that:

[TABLE]

When the inner iteration is finished and condition (17) is satisfied, run the following updates:

[TABLE]

where $y^{t}$ is the approximate solution to (5), and go back to Step 1.

The algorithm’s complexity is stated in the following proposition based on Proposition 2.2 in [10].

Proposition 6

Let $\alpha_{1},\alpha_{2}>0$ be the strong convexity parameters of $\omega_{1}(\xi)$, $\omega_{2}(p)$, $\|\cdot\|_{(1)}$ and $\|\cdot\|_{(2)}$ be two norms and $L_{uv}>0$, $u,v\in\{1,2\}$ such that:

[TABLE]

and

[TABLE]

and $\gamma_{1}$, $\gamma_{2}$ in (6) be equal to:

[TABLE]

and $\gamma_{t}$ satisfy the following condition for all iterations $t$:

[TABLE]

Under these assumptions, in each iteration $t$ there are at most two inner iterations (16) and the number $T$ of iterations to obtain an $\epsilon$-approximation of the solution to (5) is

[TABLE]

$\square$**

The cost of each iteration is $\mathcal{O}(NMS)$ – it consists of computing the derivatives of $\phi$ w.r.t. $\xi$ and $p$ and computing the proximal operator at most a fixed number of times. The total complexity depends on the dependence of the terms $\Theta_{1}$, $\alpha_{1}$ on $S$. For example, if $\mathcal{B}$ is chosen to be an $S$-dimensional Euclidean ball, the complexity is:

[TABLE]

Remark 3

In the experiments, we observe a strong dependence of the convergence speed on the parameters $\gamma^{t}$. The observed convergence is faster if $\gamma^{t}$ is larger. However, a small $\gamma^{t}$ is needed to guarantee that the number of inner iterations is at most 2. For that reason, we implemented an update rule multiplying $\gamma^{t}$ by 1.5 if the number of inner iterations was at most 2, and dividing it by 1.5 otherwise. This heuristic performed well in our computational study.

3.3 Average cost constraint

A practically relevant extension of (CC) is the situation where there is a constraint on the average cost of the input distribution [13]. In such case, the feasible set of $p$ is $\overline{\Delta}_{N}=\{p\in\mathbb{R}^{N}_{+}:\ 1^{\top}p=1,\ a^{\top}b\leq b\}$ and the worst-case channel capacity is equal to:

[TABLE]

Our goal is to solve (18) efficiently with the same machinery as the basic algorithm. Hence, we want to transform (18) to a saddle point problem where the maximization and minimization are done over compact sets for which it is still easy to compute the proximal operators. To do this, we make the following mild assumption and recall a well-known property of the average mutual information.

Assumption 1

The inner problem in (18) is strictly feasible and its holds that $\min_{n}a_{n}<b$ and $\max_{n}a_{n}>b$. $\square$

Property 1** **(See [7])

It holds that:

[TABLE]

By dualizing the inner problem in (18) w.r.t. the average cost constraint, we obtain the following problem, which by Assumption 1 and strong duality is equivalent to (18):

[TABLE]

The expression in (19) is convex in $(Q,\lambda)$ and concave in $p$. However, the domain for $\lambda$ is unbounded whereas the use of the algorithm of [10] requires compactness of the domain in both $p$ and $(Q,\lambda)$. A way to ensure it is to find a $\Lambda$ such that $\lambda^{*}\leq\Lambda$. To find such a $\Lambda$, we fix some $Q\in\mathcal{Q}$ and consider the maximized function of $\lambda$ in (19):

[TABLE]

First, we will show that $g(\lambda)$ has a shape similar to the one in the left panel of Figure 1 (convex and diverging to $+\infty$ as $\lambda\rightarrow+\infty$). This will imply that there exists a point $\Lambda>0$ such that $\partial g(\Lambda)$ contains a positive number. Thus, $g(\lambda)$ will be increasing for $\lambda>\Lambda$ and the minimizer $\lambda^{*}$ satisfies $\lambda^{*}\leq\Lambda$.

The function $g(\lambda)$ is convex as being a pointwise supremum of convex functions. For each $\lambda$ the supremum in (20) is attained because the maximized function of $p$ is continuous and $\Delta_{N}$ is compact. Next, we have the following lemma.

Lemma 1

Under Assumption 1 it holds that

[TABLE]

Proof.

See Appendix A.5. ∎

We know that $\min_{\lambda\geq 0}g(\lambda)$ is attained either at $\lambda^{*}=0$ or at a $\lambda^{*}>0$ such that $0\in\partial g(\lambda^{*})$. To find $\Lambda$ such that $g(\lambda)$ is increasing on $[\Lambda,+\infty)$ we have the following proposition.

Proposition 7

Under Assumption 1 and for $\Lambda>\log N/(b-\min_{n}a_{n})$, there exists a $d>0$ such that:

[TABLE]

Proof.

See Appendix A.6. ∎

Since $d>0$ belongs to the subdifferential $\partial g(\Lambda)$ we obtain that for $t>0$:

[TABLE]

Hence, it holds that $\lambda^{*}\leq\Lambda=\log N/(b-\min_{n}a_{n})$ and this bound is independent of $Q$. In this setup, the algorithm of [10] can be used to solve (19) – it is only necessary to define a strongly convex distance generating function $\omega_{1}(\cdot)$ for the joint vector $(\xi,\lambda)$ over the set $\mathcal{B}\times[0,\Lambda]$. For example, when $\mathcal{B}$ is a $\infty$-norm box, such a function is $\omega_{1}(\xi,\lambda)=\xi^{\top}\xi/2+\lambda^{2}/2$.

4 Numerical experiments

4.1 Binary symmetric channel

Consider the binary symmetric channel as in Section 2.3.1. We assume that the error probability belongs to an interval $\beta\in[0.15,0.45]$. In the setup of our algorithm this corresponds to the channel law matrix:

[TABLE]

We run the proximal algorithm with starting point $p^{0}=(0.9,0.1)^{\top}$ and $\xi^{0}=0$. By Proposition 4 we know that the saddle point corresponds to $p=(0.5,0.5)^{\top}$ and $\xi=1$. The right panel of Figure 1 illustrates the evolution of the decision variables over the first 100 iterations.

4.2 Impact of uncertainty

The next example illustrates the impact of uncertainty on channel capacity. Consider the deterministic channel to be a randomly generated one, with a nominal channel law matrix $Q_{0}$ with $N=M=50$ simulated randomly as:

[TABLE]

This way of sampling has two purposes: (i) keeping all the entries of $Q^{0}_{nm}\geq 10^{-5}$, (ii) reducing the number of ‘dominant’ entries per row by using the 4-th power.

We define the uncertainty as follows:

[TABLE]

where

[TABLE]

where $0\leq\Gamma\leq 1$ and $s_{n}$, $n=1,\ldots,N$ are sampled independently and uniformly from $\{1,2,\ldots,S\}$. In words, there are $S$ underlying primitive uncertainties $\xi_{s}$ and the ambiguity setup allows the perturbed channel to approach the channel in which every row of the channel law matrix is uniformly distributed, and the maximum size of perturbation is controlled by $\Gamma$. The Euclidean norm of the perturbation vector $\xi$ is restricted to be no greater than $1$, so that not all components $\xi_{s}$ can attain value 1 simultaneously. In the numerical setup we take $S=5$.

Additionally, we consider the version of this problem with additional average cost constraint $a^{\top}p\leq b$. We set $b=1$ and the $n$-component of $a$ is given by:

[TABLE]

where $\overline{p}$ is the optimal solution to the problem without the constraint. In this way, we impose a very high penalty on the 10 input symbols whose optimal probability in the problem without the linear constraint was larger than or equal to 0.05.

The left panel in Figure 2 illustrates the change in the channel capacity for the problem without and with the average cost constraint. We observe that the capacity of the channel drops significantly as the uncertainty scaling parameter $\Gamma$ increases. We see also that the capacity of the channel with the average cost constraint is strictly smaller than of the channel without such a constraint for all $\Gamma$.

4.3 Channel affected by a single uncertainty

Next we consider an experiment such that the channel is affected by a single large source of uncertainty and the idiosyncratic uncertainties of each row are negligible.

We consider an uncertain channel with $N=M>4$ where:

[TABLE]

where $\xi\in[-1,1]$ and the matrices are defined as:

[TABLE]

where $\tau=10^{-5}$ and the perturbation matrix is

[TABLE]

The nominal matrix $Q_{0}$ reflects the possible events following the choice of the $n$-th input symbol:

• •

the output symbol is the $n$-th output symbol with probability $0.92-(N-5)\tau$;

• •

the output symbol is one of the 4 nearest neighbors of the $n$-th output symbol, each with probability 0.02;

• •

other output symbol occurs, each with a small probability $\tau$.

Perturbation of the channel takes away (or adds) the probability mass from the $n$-th symbol adding it to (taking it away from) the probability masses of these four neighboring symbols. We set the vector $w$ to be parametrized by integer $0\leq W\leq m$ as follows:

[TABLE]

In this way, the first $W$ rows of the channel transition matrix are affected by uncertainty $\xi$ in a way that is opposite to the remaining $M-W$ rows. We set $M=N=50$. We run the algorithm up to accuracy 0.005. The right panel of Figure 2 illustrates the results for different values of $W$.

As one can see, the figure is symmetric because the problem is symmetric around $W=25$. For the extreme values of $W$, the capacity loss due to uncertainty amounts to $7\%$. We observe that for $W=25$ the worst-case channel capacity is equal to the capacity of a channel defined by $Q_{0}$. This is because the worst-case realization of $\xi$ at the saddle point is exactly $\xi=0$.

5 Summary

In this paper we have derived the convex robust counterparts of the channel capacity problem affected by uncertainty in the channel law matrix, and presented a first-order saddle point algorithm with $\mathcal{O}(1/T)$ convergence rate that potentially can handle large-scale problems. Our numerical examples illustrate the fact that presence of uncertainty can lead to a decrease in channel capacity.

Appendix A Proofs

A.1 Proposition 2

Proof.

Similar to the proof of Proposition 1, we use the results of Ben-Tal et al. [2015] based on Fenchel duality. Denote the feasible set for $(t,p,V)$ by $\mathcal{D}$ and define

[TABLE]

Then, we have:

[TABLE]

We first need to derive the support function $\delta^{\ast}((a,b,C)|\mathcal{D})$. We use the fact that $\mathcal{D}$ is described by the following convex functions:

[TABLE]

It holds that Ben-Tal et al. [2015]:

[TABLE]

where $h_{m}^{*}(\lambda,p,V)$, $m=1,\ldots,M+2$ are the convex conjugates of the defining functions given by:

[TABLE]

Now, we need to derive the concave conjugate $g_{\ast}(a,b,C)$. Define:

[TABLE]

Again, we know that Ben-Tal et al. [2015]:

[TABLE]

where:

[TABLE]

Combining the two results – for the support function and the concave conjugate – we obtain the following problem equivalent to (21):

[TABLE]

To simplify, we multiply the first constraint by $u_{m}$, notice that $C^{M+1}=C^{M+2}=0$, multiply the constraints in the 6-th row by $u_{M+1}$ and $u_{M+2}$, respectively, and substitute $a^{m}_{n}=Q_{nm}$ to obtain:

[TABLE]

Now we notice that:

[TABLE]

where the subsequent inequalities follow from the constraints in the 3rd, 4th and 1st rows of (22), respectively. Thanks to this, we obtain the following problem equivalent to (22):

[TABLE]

The final formulation is obtained by substituting $u_{m}=\exp(v_{m}-\mu-1)$ and minimizing w.r.t. $\mu$. Strong duality holds for the pair (P) - (D) due to the fact that Slater condition clearly holds for problem (P). ∎

A.2 Proposition 3

Proof.

It is enough to find values for the variables $\lambda$, $p$, $V$ in the primal problem (P) such that the objective in the primal problem attains the value in (4). First, take $p_{n}=1/N$ and $\lambda_{n}=(\log N)/N$ to obtain the following form of the problem:

[TABLE]

Set $V_{nm}=-(\log P_{nm})/N$ where $P^{\top}\in\Delta_{M\times N}$ (each column of $P$ belongs to a simplex). Then, the problem’s constraints hold and for the objective we have:

[TABLE]

where the penultimate equality follows from the fact that the maximizer of

[TABLE]

is given by (see Ben-Tal and Teboulle [1988])

[TABLE]

and the last equality follows by symmetry of the channel – each row is the same and the sum of entries in each column is the same. ∎

A.3 Proposition 4

Proof.

To show that this is true, we need to give a set of primal and dual variables for the binary symmetric channel such that the objective function values of the primal and dual robust problems are equal. This is achieved by the following set of values:

[TABLE]

and

[TABLE]

Inserting these values and noticing that:

[TABLE]

we conclude that the primal and dual objective values match indeed. ∎

A.4 Proposition 5

Proof.

Consider the entries of the Jacobian of $\Psi(Q,p)$ (Hessian of $f(Q,p)$). We have the following formulas, where on the right we write the upper bound on the absolute value of a given term:

[TABLE]

[TABLE]

[TABLE]

Due to the boundedness of the Jacobian, the mapping is Lipschitz continuous. ∎

A.5 Lemma 1

Proof.

Define a function $\underline{g}(\lambda)$:

[TABLE]

where $\underline{p}_{n^{*}}=1$ for $n^{*}=\min\{n:\ a_{n}=\min_{n}a_{n}\}$ and $\underline{p}_{n}=0$ otherwise. It holds that $g(\lambda)\geq\underline{g}(\lambda)$ for all $\lambda\in\mathbb{R}$. Using Property 1 we have:

[TABLE]

By Assumption 1 we have $b-\min_{n}a_{n}>0$. In this way we obtain that $\underline{g}(\lambda)$ diverges to $+\infty$ as $\lambda\rightarrow+\infty$. Since $g(\lambda)\geq\underline{g}(\lambda)$, the claim follows. ∎

A.6 Proposition 7

Proof.

First, note that:

[TABLE]

By contradiction assume that for $\lambda>\log N/(b-\min_{n}a_{n})$ it holds that all the maximizers $p$ are such that $b-\sum_{n=1}^{N}p_{n}a_{n}\leq 0$. For such such a $p$ we have the following bound on the value of $g(\lambda)$:

[TABLE]

where the last inequality follows from Property 1. Now, consider $\underline{p}_{n^{*}}=1$ for $n^{*}=\min\{n:\ a_{n}=\min_{n}a_{n}\}$ and $\underline{p}_{n}=0$ otherwise. In this case, we obtain:

[TABLE]

where the inequalities follow from Property 1 and the assumption $\lambda>\log N/(b-\min_{n}a_{n})$. We obtain a contradiction because there must be a maximizing $p$ such that $d=b-\sum_{n=1}^{N}p_{n}a_{n}>0$. ∎

Appendix B Algorithm of Nemirovski [2004] and its convergence

In this Appendix we provide a self-contained version of the algorithm of Nemirovski [2004] and the theorem stating its convergence. Consider the problem

[TABLE]

Define $z=(x,y)$, $Z=X\times Y$ in the Euclidean space and

[TABLE]

Assumption 2

The sets $X$, $Y$ are convex and compact. It holds that $\phi(x,y)\in C^{1,1}$, i.e., $\phi(x,y)$ is differentiable and $\Phi(z)$ is Lipschitz continuous:

[TABLE]

where $L>0$ and $\|\cdot\|$ is a norm. There exist functions $\omega_{1}:X\rightarrow\mathbb{R}$, $\omega_{2}:Y\rightarrow\mathbb{R}$ that are $\alpha_{1}$ and $\alpha_{2}$-strongly convex:

[TABLE]

$\square$**

Define

[TABLE]

Let $\|\cdot\|_{(1)}$ and $\|\cdot\|_{(2)}$ be two norms and $L_{uv}>0$, $u,v\in\{1,2\}$ be such that:

[TABLE]

and define the distance generating function

[TABLE]

where

[TABLE]

In this setting, under a proper norm, the following is a Lipschitz constant $L$ of $\Phi(z)$ on $Z$:

[TABLE]

Define the proximal mapping:

[TABLE]

In this setup, the algorithm is as follows.

Choose a starting point $z_{0}\in Z$. 2. 2.

Given a $z^{t-1}$ check whether

[TABLE]

If this is the case, claim that $z^{t-1}$ is the solution to (23). If not, go to 3. 3. 3.

Choose $\gamma^{t}>0$, set $w^{t,0}:=z^{t-1}$ and run the iteration

[TABLE]

until the condition

[TABLE]

is met. Denote by $s_{t}$ the corresponding value of $s$ and set $z^{t}=w^{t,s_{t}-1}$ and $z_{t}=w^{t,s_{t}}$.

In particular, when $\gamma^{t}\leq 1/\sqrt{2}L$ the number of inner iterations is at most two. Convergence of the algorithm is stated in the following theorem, based on Proposition 2.2 in Nemirovski [2004].

Theorem 1

Assume Assumption 2 holds. Define

[TABLE]

Then, it holds that:

If the algorithm terminates at a certain step $T$ according to the rule in Step 1, then $z^{T-1}$ is a solution to (23). 2. 2.

If the algorithm does not terminate in the course of $T$ steps then

[TABLE]