Kolmogorov Capacity with Overlap

TL;DR

This paper introduces a new way to measure information in uncertain systems, extending traditional concepts to handle non-stochastic noise and errors.

Contribution

The paper introduces δ-mutual information and generalizes Kolmogorov ϵ-capacity to allow for overlap and decoding errors.

Findings

δ-mutual information equals (ϵ,δ)-capacity in communication over ϵ-noise channels.

The theory extends to non-stochastic, memoryless, and stationary noise models.

The framework allows for decoding errors while maintaining worst-case guarantees.

Abstract

The notion of δ-mutual information between non-stochastic uncertain variables is introduced as a generalization of Nair’s non-stochastic information functional. Several properties of this new quantity are illustrated and used in a communication setting to show that the largest δ-mutual information between received and transmitted codewords over ϵ-noise channels equals the (ϵ,δ)-capacity. This notion of capacity generalizes the Kolmogorov ϵ-capacity to packing sets of overlap at most δ and is a variation of a previous definition proposed by one of the authors. Results are then extended to more general noise models, including non-stochastic, memoryless, and stationary channels. The presented theory admits the possibility of decoding errors, as in classical information theory, while retaining the worst-case, non-stochastic character of Kolmogorov’s approach.

Figures

Full text

1. Introduction

Shannon’s celebrated channel coding theorem states that the capacity is the supremum of the mutual information between the input and the output of the channel [1]. In this setting, the mutual information is intended as the amount of information obtained regarding the random variable at the input of the channel by observing the random variable at the output of the channel, and the capacity is the largest rate of communication that can be achieved with an arbitrarily small probability of error. In an effort to provide an analogous result for safety-critical control systems where occasional decoding errors can result in catastrophic failures, Nair introduced a non-stochastic mutual information functional and established that this equals the zero-error capacity [2], namely the largest rate of communication that can be achieved with zero probability of error. Nair’s approach is based on the calculus of non-stochastic uncertain variables (UVs), and his definition of mutual information in a non-stochastic setting is based on the quantization of the range of uncertainty of a UV induced by the knowledge of the other. While Shannon’s theorem leads to a single letter expression, Nair’s result is multi-letter, involving the non-stochastic information between codeword blocks of n symbols. The zero-error capacity can also be formulated as a graph-theoretic property, and the absence of a single-letter expression for general graphs is well known [3,4]. Extensions of Nair’s non-stochastic approach to characterize the zero-error capacity in the presence of feedback from the receiver to the transmitter using nonstochastic directed mutual information have been considered in [5].

Kolmogorov introduced the notion of $[eqn]$ -capacity in the context of functional spaces as the logarithm base two of the packing number of the space, namely the logarithm of the maximum number of balls of radius $[eqn]$ that can be placed in the space without overlap [6]. Determining this number is analogous to designing a channel codebook such that the distance between any two codewords is at least $[eqn]$ . In this way, any transmitted codeword that is subject to a perturbation of, at most, $[eqn]$ can be recovered at the receiver without error. It follows that the $[eqn]$ -capacity per transmitted symbol (viz. per signal dimension) corresponds to the zero-error capacity of an additive channel having arbitrary bounded noise of the radius at most $[eqn]$ . Lim and Franceschetti extended this concept introducing the $[eqn]$ capacity [7], defined as the logarithm base two of the largest number of balls of radius $[eqn]$ that can be placed in the space with an average codeword overlap of, at most, $[eqn]$ . In this setting, $[eqn]$ measures the amount of error that can be tolerated when designing a codebook in a non-stochastic setting, and the $[eqn]$ capacity per transmitted symbol corresponds to the largest rate of communication with error at most $[eqn]$ .

The first contribution of this paper is to consider a generalization of Nair’s mutual information based on a quantization of the range of uncertainty of a UV given the knowledge of another, which reduces the uncertainty to, at most, $[eqn]$ , and to show that this new notion corresponds to the $[eqn]$ capacity. Our definition of $[eqn]$ capacity is a variation of the one in [7], as it is required to bound the overlap between any pair of balls, rather than the average overlap. For $[eqn]$ , we recover Nair’s result for the Kolmogorov $[eqn]$ -capacity or, equivalently, for the zero-error capacity of an additive, bounded noise channel. We then extend the results to more general channels where the noise can be different across codewords and is not necessarily contained within a ball of radius $[eqn]$ . Finally, we consider the class of non-stochastic, memoryless, stationary uncertain channels, where the noise experienced by a codeword of n symbols factorizes into n identical terms describing the noise experienced by each codeword symbol. This is the non-stochastic analog of a discrete memoryless channel (DMC), where the current output symbol depends only on the current input symbol, not on any of the previous input symbols, and where the noise distribution is constant across symbol transmissions and differs from Kolmogorov’s $[eqn]$ -noise channel, where the noise experienced by one symbol affects the noise experienced by other symbols (in Kolmogorov’s setting, the noise occurs within a ball of radius $[eqn]$ . It follows that for any realization where the noise along one dimension (viz. symbol) is close to $[eqn]$ , the noise experienced by all other symbols lying in the remaining dimensions must be close to zero.). Letting $[eqn]$ be the confidence of correct decoding after transmitting n symbols, we introduce several notions of capacity and establish coding theorems in terms of mutual information for all of them, including a generalization of the zero-error capacity that requires the error sequence $[eqn]$ to remain constant and a non-stochastic analog of Shannon’s capacity that requires the error sequence to vanish, as in $[eqn]$ .

Finally, since in Nair’s case, all of our results are multi-letter, in the Supplementary Materials, we provide some sufficient conditions for the factorization of the mutual information leading to a single-letter expression for the non-stochastic capacity of stationary, memoryless, uncertain channels, provide some examples in which these conditions are satisfied, and compute the corresponding capacity.

The rest of the paper is organized as follows: Section 2 introduces the mathematical framework of non-stochastic uncertain variables that are used throughout the paper. Section 3 introduces the concept of non-stochastic mutual information. Section 4 gives an operational definition of the capacity of a communication channel and relates it to the mutual information. Section 5 extends the results to more general channel models, and Section 6 concentrates on the special case of stationary, memoryless, uncertain channels. Section 7 draws conclusions and discusses future directions.

2. Uncertain Variables

We start by reviewing the mathematical framework used in [2] to describe UVs. A UV X is a mapping from a sample space $[eqn]$ to a set $[eqn]$ , i.e., for all $[eqn]$ , we have $[eqn]$ , namely

$$[eqn]$$

Given a UV X, the marginal range of X is

$$[eqn]$$

The joint range of the two UVs X and Y is

$$[eqn]$$

Given a UV Y, the conditional range of X given $[eqn]$ is

$$[eqn]$$

and the conditional range of X given Y is

$$[eqn]$$

Thus, $[eqn]$ denotes the uncertainty in X, given the realization of Y and that $[eqn]$ represents the total joint uncertainty of X and Y, namely

$$[eqn]$$

Finally, two UVs, X and Y, are independent if for all $[eqn]$ ,

$$[eqn]$$

which also implies that for all $[eqn]$ ,

$$[eqn]$$

3. δ-Mutual Information

3.1. Uncertainty Function

We now introduce a class of functions that are used to express the amount of uncertainty in determining one UV given another. In our setting, an uncertainty function associates a positive number with a given set, which expresses the “massiveness” or “size” of that set.

Definition 1.  Given the set of non-negative real numbers $[eqn]$ and any set of $[eqn]$ , $[eqn]$ is an uncertainty function if it is finite and strongly transitive:

*We have $[eqn]$ , and for all $[eqn]$ *

$$[eqn]$$

For all $[eqn]$ we have

$$[eqn]$$

In the case where $[eqn]$ is measurable, an uncertainty function can easily be constructed using a measure. In the case where $[eqn]$ is a bounded (not necessarily measurable) metric space and the input set $[eqn]$ contains at least two points, an example of an uncertainty function is the diameter.

3.2. Association and Dissociation Between UVs

We now introduce notions of association and dissociation between UVs. In the following definitions, we let $[eqn]$ and $[eqn]$ be uncertainity functions defined over sets $[eqn]$ and $[eqn]$ corresponding to UVs X and Y. We use the notation $[eqn]$ to indicate that for all $[eqn]$ , we have $[eqn]$ . Similarly, we use $[eqn]$ to indicate that for all $[eqn]$ , we have $[eqn]$ , where $[eqn]$ and $[eqn]$ . For $[eqn]$ , we assume that $[eqn]$ is always satisfied, while $[eqn]$ is not. Whenever we consider $[eqn]$ , we also assume that $[eqn]$ and $[eqn]$ , where $[eqn]$ and $[eqn]$ .

Definition 2.  The sets of association for UVs X and Y are

$$[eqn]$$ $$[eqn]$$

The sets of association are used to describe the correlation between two uncertain variables. Since $[eqn]$ , the exclusion of the zero value in (11) and (12) occurs when there is no overlap between the two conditional ranges.

Definition 3.  For any $[eqn]$ , UVs X and Y are disassociated at levels $[eqn]$ if the following inequalities hold:

$$[eqn]$$ $$[eqn]$$

and, in this case, we write $[eqn]$ .

Having UVs X and Y be disassociated at levels $[eqn]$ indicates that at least two conditional ranges $[eqn]$ and $[eqn]$ have non-zero overlap and that, given any two conditional ranges, either they do not overlap or the uncertainty associated with their overlap is greater than a $[eqn]$ fraction of the total uncertainty associated with $[eqn]$ ; the same holds for conditional ranges $[eqn]$ and $[eqn]$ and for level $[eqn]$ . The levels of disassociation can be viewed as lower bounds in the amount of residual uncertainty in each variable when the other is known. If X and Y are independent, then all the conditional ranges completely overlap, $[eqn]$ and $[eqn]$ contain only the element one, and the variables are maximally disassociated (see Figure 1a).

In this case, knowledge of Y does not reduce the uncertainty of X, and vice versa. On the other hand, when the uncertainty associated with any of the non-zero intersections of the conditional ranges decreases but remains positive, X and Y become less disassociated in the sense that knowledge of Y can reduce the residual uncertainty of X, and vice versa (see Figure 1b). When the intersection between every pair of conditional ranges becomes empty, the variables cease to be disassociated (see Figure 1c). Note that excluding the value of 1 in the definition of disassociation allows us to distinguish the case of disassociation from the case of full independence.

An analogous definition of association is given to provide upper bounds on the residual uncertainty of one uncertain variable when the other is known.

Definition 4.  For any $[eqn]$ , we say that UVs X and Y are associated at levels $[eqn]$ if the following inequalities hold:

$$[eqn]$$ $$[eqn]$$

and in this case, we write $[eqn]$ .

Note that $[eqn]$ is included in Definition 4 and not in Definition 3. This is because in Definition 3, we have a strict lower bound on the uncertainty in the association sets.

The following lemma provides the necessary and sufficient conditions for association to hold at given levels. These conditions are stated for all points in the marginal ranges $[eqn]$ and $[eqn]$ . They show that in the case of association, one can also include in the definition the conditional ranges that have zero intersection. This is not the case for disassociation.

Lemma 1.  For any $[eqn]$ , $[eqn]$ if and only if for all $[eqn]$ , we have

$$[eqn]$$

and for all $[eqn]$ , we have

$$[eqn]$$

Proof. The proof is given in Appendix A. □

An immediate yet important consequence of our definitions is that both association and disassociation at given levels $[eqn]$ cannot hold simultaneously. We also understand that, given any two UVs, one can always select $[eqn]$ and $[eqn]$ to be large enough such that they are associated at levels $[eqn]$ . In contrast, as the smallest value in the sets $[eqn]$ and $[eqn]$ tends towards zero, the variables eventually cease to be disassociated. Finally, it is possible that two uncertain variables are neither associated nor disassociated at given levels $[eqn]$ . Also, any two uncertain variables are associated at level $[eqn]$ trivially by definition.

Example 1.  Consider three individuals, a, b, and c, are going for a walk along a path. Assume they take, at most, 15, 20, and 10 min to finish their walk, respectively. Assume a starts walking at time 5:00, b starts walking at 5:10, and c starts walking at 5:20. Figure 2 shows the possible time intervals for the walkers on the path. Let an uncertain variable W represent the set of walkers that are present on the path at any time and an uncertain variable T represent the time at which any walker on the path finishes their walk. Then, we have the following marginal ranges:

$$[eqn]$$ $$[eqn]$$

We also have the following conditional ranges:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

For all t ∈ [5:00, 5:10), we have

$$[eqn]$$

for all t ∈ [5:10, 5:15], we have

$$[eqn]$$

for all t ∈ (5:15, 5:20), we have

$$[eqn]$$

and for all t ∈ [5:20, 5:30], we have

$$[eqn]$$

Now, let the uncertainty function of a time set $[eqn]$ be

$$[eqn]$$

where $[eqn]$ is the Lebesgue measure. Let the uncertainty function $[eqn]$ associated with a set of individuals be the cardinality of the set. Then, the sets of association are

$$[eqn]$$ $$[eqn]$$

It follows that for all $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$

and the residual uncertainty in W given T is at least a $[eqn]$ fraction of the total uncertainty in W, while the residual uncertainty in T given W is at least a $[eqn]$ fraction of the total uncertainty in T. On the other hand, for all $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$

and the residual uncertainty in W given T is at most a $[eqn]$ fraction of the total uncertainty in W, while the residual uncertainty in T given W is at most a $[eqn]$ fraction of the total uncertainty in T.

Finally, if $[eqn]$ or $[eqn]$ , then W and T are neither associated nor disassociated.

3.3. δ-Mutual Information

We now introduce the mutual information between uncertain variables in terms of some structural properties of covering sets. Intuitively, for any $[eqn]$ the $[eqn]$ -mutual information, expressed in bits, represents the most refined knowledge that one uncertain variable provides about the other at a given level of confidence $[eqn]$ . We express this idea by considering the quantization of the range of uncertainty of one variable, induced by the knowledge of the other. Such quantization ensures that the variable can be identified with uncertainty at most $[eqn]$ . The notions of association and disassociation introduced above are used to ensure that the mutual information is well defined, in that it can be positive and exhibits a certain symmetric property.

Definition 5.  δ-Connectedness and δ-isolation.

• For any $[eqn]$ , points $[eqn]$ are δ-connected via $[eqn]$ and are denoted by $[eqn]$ if there exists a finite sequence $[eqn]$ of conditional sets such that $[eqn]$ , $[eqn]$ and for all $[eqn]$ , we have

$$[eqn]$$

If $[eqn]$ and $[eqn]$ , then we say that $[eqn]$ and $[eqn]$ are singly δ-connected via $[eqn]$ , i.e., there exists a y such that $[eqn]$ .

• A set $[eqn]$ is δ-connected via $[eqn]$ if every pair of points in the set is δ-connected via $[eqn]$ .
• A set $[eqn]$ is singly δ-connected via $[eqn]$ if there exists a $[eqn]$ such that every point in the set is contained in $[eqn]$ , namely $[eqn]$ .
• Two sets $[eqn]$ are δ-isolated via $[eqn]$ if no point in $[eqn]$ is δ-connected to any point in $[eqn]$ .

Example 2.  *Consider the same setting discussed in Example 1. For $[eqn]$ , two points at times 5:05 and 5:25 ∈ 〚T〛 are δ-connected. The sequence of conditional sets connecting the two points is $[eqn]$ , where the sets are defined in (21) and (22). This is because 5:05 ∈ 〚T|{a}〛, 5:25 ∈ 〚T|{b}〛 and *

$$[eqn]$$

For all $[eqn]$ , two points at times 5:00 and 5:05 $[eqn]$ are singly δ-connected since 5:00, 5:05 $[eqn]$ .

Likewise, for all $[eqn]$ , the set $[eqn]$ is singly δ-connected by definition.

For $[eqn]$ , the set $[eqn]$ is δ-connected. This is because for all $[eqn]$ , one of the following scenarios holds:

• • $[eqn]$ .*
• • $[eqn]$ .*
• • $[eqn]$ and $[eqn]$ , or vice-versa.*

In the first two scenarios, the points $[eqn]$ and $[eqn]$ are singly δ-connected. In the third scenario, the points are δ-connected since

$$[eqn]$$

For all $[eqn]$ , the two sets $[eqn]$ and $[eqn]$ are δ-isolated since there is no overlap between the two sets.

Definition 6.  δ-overlap family.

For any $[eqn]$ , a $[eqn]$ δ-overlap family of $[eqn]$ , denoted by $[eqn]$ , is the largest family of distinct sets covering $[eqn]$ , such that

1. Each set in the family is δ-connected and contains at least one singly δ-connected set of the form $[eqn]$ .
2. The measure of overlap between any two distinct sets in the family is at most $[eqn]$ , namely for all $[eqn]$ , such that $[eqn]$ ; we also have $[eqn]$ .
3. For every singly δ-connected set, there exist a set in the family containing it.

The first property of the $[eqn]$ -overlap family ensures that points in the same set of the family cannot be distinguished with confidence of at least $[eqn]$ , while also ensuring that each set cannot be arbitrarily small. The second and third properties ensure that points that are not covered by the same set of the family can be distinguished with confidence of at least $[eqn]$ . It follows that the cardinality of the covering family represents the most refined knowledge at a given level of confidence $[eqn]$ that we can have about X, given the knowledge of Y. This also corresponds to the most refined quantization of the set $[eqn]$ induced by Y. This interpretation is analogous to the one in [2], extending the concept of overlap partition introduced there to a $[eqn]$ -overlap family in this work. The stage is now set to introduce the $[eqn]$ -mutual information in terms of the $[eqn]$ -overlap family.

Definition 7.  The δ-mutual information provided by Y about X is

$$[eqn]$$

if a $[eqn]$ δ-overlap family of $[eqn]$ exists; otherwise, it is zero.

We now show that when variables are associated at level $[eqn]$ , there exists a $[eqn]$ -overlap family, so that the mutual information is well defined.

Theorem 1.  If $[eqn]$ , then there exists a δ-overlap family $[eqn]$ .

Proof. We show that

$$[eqn]$$

satisfies all the three properties of $[eqn]$ -overlap family in Definition 6. First, note that $[eqn]$ is a cover of $[eqn]$ , since $[eqn]$ , even though $[eqn]$ for different y may overlap with each other. Second, each set in the family $[eqn]$ is singly $[eqn]$ -connected via $[eqn]$ , since trivially, any two points $[eqn]$ are singly $[eqn]$ -connected via the same set. It follows that Property 1 of Definition 6 holds.Now, since $[eqn]$ , then by Lemma 1, for all $[eqn]$ , we have

$$[eqn]$$

which shows that Property 2 of Definition 6 holds. Finally, it is also easy to see that Property 3 of Definition 6 holds, since $[eqn]$ contains all sets $[eqn]$ . Hence, $[eqn]$ satisfies all the three properties in Definition 6, which implies that there exists at least one set satisfying these conditions. Hence, the maximum over these sets is defined and the claim follows. □

Next, we show that a $[eqn]$ -overlap family also exists when variables are disassociated at level $[eqn]$ . In this case, we also characterize the mutual information in terms of a partition of $[eqn]$ .

Definition 8.  δ-isolated partition.

A $[eqn]$ δ-isolated partition of $[eqn]$ , denoted by $[eqn]$ , is a partition of $[eqn]$ such that any two sets in the partition are δ-isolated via $[eqn]$ .

Theorem 2.  If $[eqn]$ , then the following holds:

• 1.  There exists a unique δ-overlap family $[eqn]$ .
• 2.  The δ-overlap family is the δ-isolated partition of largest cardinality, in that, for any $[eqn]$ , we have

$$[eqn]$$

where the equality holds if and only if $[eqn]$ .

Proof. First, we show the existence of a $[eqn]$ -overlap family. For all $[eqn]$ , let $[eqn]$ be the set of points that are $[eqn]$ -connected to x via $[eqn]$ , namely

$$[eqn]$$

Then, we let

$$[eqn]$$

and show that this is a $[eqn]$ -overlap family. First, note that since $[eqn]$ , we know that $[eqn]$ is a cover of $[eqn]$ . Second, for all $[eqn]$ , there exists a $[eqn]$ such that $[eqn]$ , and since any two points $[eqn]$ are singly $[eqn]$ -connected via $[eqn]$ , we understand that $[eqn]$ . It follows that every set in the family $[eqn]$ contains at least one singly $[eqn]$ -connected set. For all $[eqn]$ , we also have $[eqn]$ and $[eqn]$ . Since $[eqn]$ , by Lemma A2 in Appendix C, this implies that $[eqn]$ . It follows that every set in the family $[eqn]$ is $[eqn]$ -connected and contains at least one singly $[eqn]$ -connected set, and we conclude that Property 1 of Definition 6 is satisfied.We now claim that for all $[eqn]$ , if

$$[eqn]$$

then

$$[eqn]$$

This can be proven by contradiction. Let $[eqn]$ and assume that $[eqn]$ . By (9), this implies that $[eqn]$ . We can then select $[eqn]$ , such that we have $[eqn]$ and $[eqn]$ . Since $[eqn]$ , by Lemma A2 in Appendix C, this also implies that $[eqn]$ , and, therefore, $[eqn]$ , which is a contradiction. It follows that if $[eqn]$ , then we must have $[eqn]$ , and, therefore,

$$[eqn]$$

We conclude that Property 2 of Definition 6 is satisfied.Finally, we understand that for any singly $[eqn]$ -connected set $[eqn]$ , there exist an $[eqn]$ such that $[eqn]$ , which by, (42), implies that $[eqn]$ . Namely, for every singly $[eqn]$ -connected set, there exist a set in the family containing it. We can then conclude that $[eqn]$ satisfies all the properties of a $[eqn]$ -overlap family.Next, we show that $[eqn]$ is a unique $[eqn]$ -overlap family, which implies that this is also the largest set satisfying the three conditions in Definition 6. By contradiction, consider another $[eqn]$ -overlap family $[eqn]$ . For all $[eqn]$ , let $[eqn]$ denote a set in $[eqn]$ containing x. Then, using the definition of $[eqn]$ and the fact that $[eqn]$ is $[eqn]$ -connected, it follows that

$$[eqn]$$

Next, we show that for all $[eqn]$ , we also have

$$[eqn]$$

from which, we conclude that $[eqn]$ .The proof of (48) is also obtained by contradiction. Assume there exists a point $[eqn]$ . Since both x and $[eqn]$ are contained in $[eqn]$ , $[eqn]$ . Let $[eqn]$ be a point in a singly connected set that is contained in $[eqn]$ , namely $[eqn]$ . Since both x and $[eqn]$ are in $[eqn]$ , we understand that $[eqn]$ . Since $[eqn]$ , we can apply Lemma A2 in Appendix C to conclude that $[eqn]$ . It follows that there exists a sequence of conditional ranges $[eqn]$ such that $[eqn]$ and $[eqn]$ , which satisfies (35). Since $[eqn]$ is in both $[eqn]$ and $[eqn]$ , we obtain $[eqn]$ , and since $[eqn]$ , we obtain

$$[eqn]$$

Without loss of generality, we can then assume that the last element of our sequence is $[eqn]$ . By Property 3 of Definition 6, every conditional range in the sequence must be contained in some set of the $[eqn]$ -overlap family $[eqn]$ . Since $[eqn]$ and $[eqn]$ , it follows that there exist two consecutive conditional ranges along the sequence and two sets of the $[eqn]$ -overlap family covering them, such that $[eqn]$ , $[eqn]$ , and $[eqn]$ . Then, we have

$$[eqn]$$

where $[eqn]$ follows from (10) and $[eqn]$ follows from (35). It follows that

$$[eqn]$$

and Property 2 of Definition 6 is violated. Thus, $[eqn]$ does not exists, which implies $[eqn]$ . Combining (47) and (48), we conclude that the $[eqn]$ -overlap family $[eqn]$ is unique.We now turn to the proof of the second part of the theorem. Since by (46), the uncertainty associated with the overlap between any two sets of the $[eqn]$ -overlap family $[eqn]$ is zero, it follows that $[eqn]$ is also a partition.Now, we show that $[eqn]$ is also a $[eqn]$ -isolated partition. This can be proven by contradiction. Assume that $[eqn]$ is not a $[eqn]$ -isolated partition. Then, there exists two distinct sets $[eqn]$ such that $[eqn]$ and $[eqn]$ are not $[eqn]$ -isolated. This implies that there exists a point $[eqn]$ and $[eqn]$ such that $[eqn]$ . Using the fact that $[eqn]$ and $[eqn]$ are $[eqn]$ -connected and Lemma A2 in Appendix C, this implies that all points in the set $[eqn]$ are $[eqn]$ -connected to all points in the set $[eqn]$ . Now, let $[eqn]$ and $[eqn]$ be points in a singly $[eqn]$ -connected set contained in $[eqn]$ and $[eqn]$ , respectively: $[eqn]$ and $[eqn]$ . Since $[eqn]$ , there exists a sequence of conditional ranges $[eqn]$ satisfying (35), such that $[eqn]$ and $[eqn]$ . Without loss of generality, we can assume $[eqn]$ and $[eqn]$ . Since $[eqn]$ is a partition, we understand that $[eqn]$ and $[eqn]$ . It follows that there exist two consecutive conditional ranges along the sequence $[eqn]$ and two sets of the $[eqn]$ -overlap family $[eqn]$ covering them, such that $[eqn]$ and $[eqn]$ and that $[eqn]$ . Similarly to (50), we hold that

$$[eqn]$$

and Property 2 of Definition 6 is violated. Thus, $[eqn]$ and $[eqn]$ do not exist, which implies that $[eqn]$ is a $[eqn]$ -isolated partition.Let $[eqn]$ be any other $[eqn]$ -isolated partition. We wish to show that $[eqn]$ and that the equality holds if and only if $[eqn]$ . First, note that every set $[eqn]$ can intersect, at most, one set in $[eqn]$ ; otherwise, the sets in $[eqn]$ would not be $[eqn]$ -isolated. Second, since $[eqn]$ is a cover of $[eqn]$ , every set in $[eqn]$ must be intersected by at least one set in $[eqn]$ . It follows that

$$[eqn]$$

Now, assume the equality holds. In this case, there is a one-to-one correspondence $[eqn]$ , such that for all $[eqn]$ , we have $[eqn]$ , and since both $[eqn]$ and $[eqn]$ are partitions of $[eqn]$ , it follows that $[eqn]$ . Conversely, assuming that $[eqn]$ , then $[eqn]$ follows trivially. □

We have introduced the notion of mutual information from Y to X in terms of the conditional range $[eqn]$ . Since, in general, we have $[eqn]$ , one may expect the definition of mutual information to be asymmetric in its arguments. Namely, the amount of information provided about X by the knowledge of Y may not be the same as the amount of information provided about Y by the knowledge of X. Although this is true in general, we show that for disassociated UVs, symmetry is retained, provided that when swapping X with Y, one also rescales $[eqn]$ appropriately. The following theorem establishes the symmetry in the mutual information under the appropriate scaling of the parameters $[eqn]$ and $[eqn]$ . The proof requires the introduction of the notions of taxicab connectedness, taxicab family, and taxicab partition, which are given in Appendix C.1, along with the proof of the theorem.

Theorem 3.  If $[eqn]$ and a $[eqn]$ -taxicab family of $[eqn]$ exists, then we have

$$[eqn]$$

4. (ϵ,δ)-Capacity

We now give a definition of the capacity of a communication channel and relate it to the notion of mutual information between the UVs introduced above. We consider a normed space $[eqn]$ to be totally bounded if, for every $[eqn]$ , $[eqn]$ can be covered by a finite number of open balls of radius $[eqn]$ . We let $[eqn]$ be a totally bounded, normed space such that for all $[eqn]$ , we have $[eqn]$ , where $[eqn]$ represents the norm. This normalization is for convenience in the notation process, and all results can easily be extended to metric spaces of any bounded norm. Let $[eqn]$ be a discrete set of points in the space, which represents a codebook.

Definition 9.  ϵ-perturbation channel.

A channel is called ϵ-perturbation if for any transmitted codeword $[eqn]$ , x is received with noise perturbation at most ϵ. Namely, we receive a point in the set

$$[eqn]$$

Given the codebook $[eqn]$ is transmitted over an $[eqn]$ -perturbation channel, all received codewords lie in the set $[eqn]$ , where $[eqn]$ . Transmitted codewords can be decoded correctly as long as the corresponding uncertainty sets at the receiver do not overlap. This can be achieved by simply associating the received codeword to the point in the codebook that is closest to it.

For any $[eqn]$ ∈ $[eqn]$ , we now let

$$[eqn]$$

where $[eqn]$ is an uncertainty function defined over the space $[eqn]$ . We also assume without loss of generality that the uncertainty associated with the whole space $[eqn]$ of received codewords is $[eqn]$ . Finally, we let $[eqn]$ be the smallest uncertainty set corresponding to a transmitted codeword, namely $[eqn]$ , where $[eqn]$ . The quantity $[eqn]$ can be viewed as the confidence we have in not confusing $[eqn]$ and $[eqn]$ in any transmission or, equivalently, as the amount of adversarial effort required to induce a confusion between the two codewords. For example, if the uncertainty function is constructed using a measure, then all the erroneous codewords generated by an adversary to decode $[eqn]$ instead of $[eqn]$ must lie inside the equivocation set depicted in Figure 3, whose relative size is given by (56). The smaller the equivocation set is, the larger the effort required by the adversary to induce an error must be. If the uncertainty function represents the diameter of the set, then all the erroneous codewords generated by an adversary to decode $[eqn]$ instead of $[eqn]$ will be close to each other in the sense of (56). Once again, the closer the possible erroneous codewords are, the harder it must be for the adversary to generate an error, since any small deviation allows the decoder to correctly identify the transmitted codeword.

We now introduce the notion of a distinguishable codebook, ensuring that every codeword cannot be confused with any other codeword, rather than with a specific one, at a given level of confidence.

Definition 10. 

• $[eqn]$ -distinguishable codebook.*

For any $[eqn]$ , $[eqn]$ , a codebook $[eqn]$ is $[eqn]$ -distinguishable if for all $[eqn]$ , we have $[eqn]$ .

For any $[eqn]$ -distinguishable codebook $[eqn]$ and $[eqn]$ , we let

$$[eqn]$$

It now follows from Definition 10 that

$$[eqn]$$

and each codeword in an $[eqn]$ -distinguishable codebook can be decoded correctly with confidence of at least $[eqn]$ . Definition 10 guarantees even more, namely that the confidence of not confusing any pair of codewords is uniformly bounded by $[eqn]$ . This stronger constraint implies that we cannot “balance” the error associated with a codeword transmission by allowing some decoding pair to have a lower confidence and enforcing other pairs to have higher confidence. This is the main difference between our definition and the one used in [7], which bounds the average confidence and allows us to relate the notion of capacity to the mutual information between pairs of codewords.

Definition 11. 

• $[eqn]$ -capacity.*

For any totally bounded, normed metric space $[eqn]$ , $[eqn]$ , $[eqn]$ , and the $[eqn]$ -capacity of $[eqn]$ is

$$[eqn]$$

where $[eqn]$ is the set of $[eqn]$ -distinguishable codebooks.

The $[eqn]$ -capacity represents the largest number of bits that can be communicated by using any $[eqn]$ -distinguishable codebook. The corresponding geometric picture is illustrated in Figure 4. For $[eqn]$ , our notion of capacity reduces to Kolmogorov’s $[eqn]$ -capacity, which is the logarithm of the packing number of the space with balls of radius $[eqn]$ .

In the definition of capacity, we have restricted $[eqn]$ to rule out cases when the decoding error can be at least as large as the error introduced by the channel and when the $[eqn]$ -capacity is infinite. Also, note that $[eqn]$ since $[eqn]$ and (10) holds.

We now relate our operational definition of capacity to the notion of UVs and mutual information introduced in Section 3. Let X be the UV corresponding to the transmitted codeword. This is a map $[eqn]$ and $[eqn]$ . Likewise, let Y be the UV corresponding to the received codeword. This is a map $[eqn]$ and $[eqn]$ . For our $[eqn]$ -perturbation channel, these UVs are such that for all $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$ $$[eqn]$$

(see Figure 5). Clearly, the set in (60) is continuous, while the set in (61) is discrete.

To measure the levels of association and disassociation between X and Y, we use an uncertainty function $[eqn]$ defined over $[eqn]$ and $[eqn]$ defined over $[eqn]$ . We introduce the feasible set

$$[eqn]$$

representing the set of UVs X such that the marginal range $[eqn]$ is a discrete set representing a codebook, and the UV can either achieve $[eqn]$ levels of disassociation or $[eqn]$ levels of association with Y. In our channel model, this feasible set also depends on the $[eqn]$ -perturbation through (60) and (61).

We can now state the non-stochastic channel coding theorem for our $[eqn]$ -perturbation channel.

Theorem 4.  For any totally bounded, normed metric space $[eqn]$ , ϵ-perturbation channel satisfying (60) and (61), $[eqn]$ , and $[eqn]$ , we have

$$[eqn]$$

Proof. First, we show that there exists a UV X and $[eqn]$ such that $[eqn]$ , which implies that the supremum is well defined. Second, for all X and $[eqn]$ such that

$$[eqn]$$

and

$$[eqn]$$

we show that

$$[eqn]$$

Finally, we show the existence of $[eqn]$ and $[eqn]$ such that $[eqn]$ .Let us begin with the first step. Consider a point $[eqn]$ . Let X be a UV such that

$$[eqn]$$

Then, we hold that the marginal range of the UV Y corresponding to the received variable is

$$[eqn]$$

and, therefore, for all $[eqn]$ , we have

$$[eqn]$$

Using Definition 2 and (67), we hold that

$$[eqn]$$

because $[eqn]$ consists of a single point, and, therefore, the set in (12) is empty.On the other hand, using Definition 2 and (69), we have

$$[eqn]$$

Using (70), and since $[eqn]$ holds for $[eqn]$ , we have

$$[eqn]$$

Similarly, using (71), we have

$$[eqn]$$

Now, combining (72) and (73), we have

$$[eqn]$$

Letting $[eqn]$ , this implies that $[eqn]$ and the first step of the proof is complete.To prove the second step, we define the set of discrete UVs

$$[eqn]$$

which is a larger set than the one containing all UVs X that are $[eqn]$ associated with Y. Now, we will show that if a UV $[eqn]$ , then the corresponding codebook $[eqn]$ . If $[eqn]$ , then there exists a $[eqn]$ such that for all $[eqn]$ , we have

$$[eqn]$$

It follows that for all $[eqn]$ , we have

$$[eqn]$$

Using $[eqn]$ , (60), $[eqn]$ and $[eqn]$ , for all $[eqn]$ , we have

$$[eqn]$$

where $[eqn]$ follows from $[eqn]$ . Putting things together, it follows that

$$[eqn]$$

Consider now a pair of X and $[eqn]$ such that $[eqn]$ and

$$[eqn]$$

If $[eqn]$ , then, using Lemma A1 in Appendix C, there exist two UVs, $[eqn]$ and $[eqn]$ and $[eqn]$ , such that

$$[eqn]$$

and

$$[eqn]$$

On the other hand, if $[eqn]$ , then (81) and (82) also trivially hold. It then follows that (81) and (82) hold for all $[eqn]$ . We now have

$$[eqn]$$

where $[eqn]$ follows from (81) and (82), $[eqn]$ follows from Lemma A3 in Appendix C since $[eqn]$ , $[eqn]$ follows by defining the codebook $[eqn]$ corresponding to the UV $[eqn]$ , and $[eqn]$ follows from the fact that using (81) and Lemma 1 allows $[eqn]$ , which implies for (79) that $[eqn]$ .Finally, let

$$[eqn]$$

which achieves the capacity $[eqn]$ . Let $[eqn]$ be the UV whose marginal range corresponds to the codebook $[eqn]$ . It follows that for all $[eqn]$ , we have

$$[eqn]$$

which implies that $[eqn]$ ,

$$[eqn]$$

Letting $[eqn]$ , and using Lemma 1, we hold that $[eqn]$ , which implies that $[eqn]$ , and the proof is complete. □

Theorem 4 characterizes the capacity as the supremum of the mutual information over all UVs in the feasible set. The following theorem shows that the same characterization is obtained if we optimize the right-hand side in (63) over all UVs in the space. It follows by Theorem 4 that rather than optimizing over all UVs representing all the codebooks in the space, a capacity-achieving codebook can be found within the smaller class $[eqn]$ of feasible sets with error at most $[eqn]$ , since for all $[eqn]$ , $[eqn]$ .

Theorem 5.  The $[eqn]$ -capacity in (63) can also be written as

$$[eqn]$$

Proof. Consider a UV $[eqn]$ , where Y is the corresponding UV at the receiver. The idea of the proof is to show the existence of a UV $[eqn]$ and the corresponding UV $[eqn]$ at the receiver, and

$$[eqn]$$

such that the cardinality of the overlap partitions

$$[eqn]$$

Let the cardinality

$$[eqn]$$

By Property 1 of Definition 6, we hold that for all $[eqn]$ , there exists an $[eqn]$ such that $[eqn]$ . Now, consider another UV $[eqn]$ whose marginal range is composed of K elements of $[eqn]$ , namely

$$[eqn]$$

Let $[eqn]$ be the UV corresponding to the received variable. Using the fact that for all $[eqn]$ , we have $[eqn]$ since (60) holds, and using Property 2 of Definition 6, for all $[eqn]$ , we obtain

$$[eqn]$$

where $[eqn]$ follows from the fact that $[eqn]$ using (91). Then, for all $[eqn]$ , we hold that

$$[eqn]$$

since $[eqn]$ . Then, by Lemma 1, it follows that

$$[eqn]$$

Since $[eqn]$ , we have

$$[eqn]$$

Therefore, $[eqn]$ and $[eqn]$ . We now hold that

$$[eqn]$$

where $[eqn]$ follows by applying Lemma A4 in Appendix C using (94) and (95) and $[eqn]$ follows from (90) and (91). Combining (96) with Theorem 4, the proof is complete. □

We now make some considerations with respect to previous results in the literature. First, we note that for $[eqn]$ , all of our definitions reduce to Nair’s ones and Theorem 4 recovers Nair’s coding theorem ([2] (Theorem 4.1)) for the zero-error capacity of an additive $[eqn]$ -perturbation channel.

Second, we point out that the $[eqn]$ -capacity considered in [7] defines the set of $[eqn]$ -distinguishable codewords such that the average overlap among all codewords is at most $[eqn]$ . In contrast, our definition requires the overlap for each pair of codewords to be at most $[eqn]$ . The following theorem provides the relationship between our $[eqn]$ and the capacity $[eqn]$ considered in [7], which is defined using the Euclidean norm.

Theorem 6.  Let $[eqn]$ be the $[eqn]$ -capacity defined in [7]. We have

$$[eqn]$$

and

$$[eqn]$$

Proof. For every codebook $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$

Since $[eqn]$ , this implies that for all $[eqn]$ , we have

$$[eqn]$$

For all $[eqn]$ , the average overlap defined in ([7] (53)) is

$$[eqn]$$

Then, we have

$$[eqn]$$

where $[eqn]$ follows from (100). Thus, we have

$$[eqn]$$

and (97) follows.Now, let $[eqn]$ be a codebook with average overlap at most $[eqn]$ , namely

$$[eqn]$$

This implies that for all $[eqn]$ , we have

$$[eqn]$$

where $[eqn]$ follows from the fact that $[eqn]$ . Thus, we have

$$[eqn]$$

and (98) follows. □

To better understand the relationship between the two capacities and show how they can be distinct, consider the case in which the output space is the union of the three $[eqn]$ -balls depicted in Figure 6; this is the only feasible output configuration.

We now compute the two capacities $[eqn]$ and $[eqn]$ in this case. We have

$$[eqn]$$

and the average overlap (101) is

$$[eqn]$$

It follows that

$$[eqn]$$

On the other hand, the worst case overlap is

$$[eqn]$$

and it follows that

$$[eqn]$$

5. (N,δ)-Capacity of General Channels

We now extend our results to more general channels where the noise can be different across codewords and is not necessarily contained within a ball of radius $[eqn]$ .

Let $[eqn]$ be a discrete set of points in the space, which represents a codebook. Any point $[eqn]$ represents a codeword that can be selected at the transmitter, sent over the channel, and received with perturbation. A channel with transition mapping $[eqn]$ associates with any point in $[eqn]$ a set in $[eqn]$ , such that the received codeword lies in the set

$$[eqn]$$

Figure 7 illustrates possible uncertainty sets associated with three different codewords.

All received codewords lie in the set $[eqn]$ , where $[eqn]$ . For any $[eqn]$ ∈ $[eqn]$ , we now let

$$[eqn]$$

where $[eqn]$ is an uncertainty function defined over $[eqn]$ . We also assume without loss of generality that the uncertainty associated with the space $[eqn]$ of received codewords is $[eqn]$ . We also let $[eqn]$ , where $[eqn]$ . Thus, $[eqn]$ is the set corresponding to the minimum uncertainty introduced by the noise mapping N.

Definition 12. 

• $[eqn]$ -distinguishable codebook.*

For any $[eqn]$ , a codebook $[eqn]$ is $[eqn]$ -distinguishable if for all $[eqn]$ , we have $[eqn]$ .

Definition 13. 

• $[eqn]$ -capacity.*

For any totally bounded, normed metric space $[eqn]$ , channel with transition mapping N, and $[eqn]$ , the $[eqn]$ -capacity of $[eqn]$ is

$$[eqn]$$

where $[eqn]$ .

We now relate our definition of capacity to the notion of UVs and mutual information introduced in Section 3. As usual, let X be the UV corresponding to the transmitted codeword and Y be the UV corresponding to the received codeword. For a channel with transition mapping N, these UVs are such that for all $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$ $$[eqn]$$

To measure the levels of association and disassociation between UVs X and Y, we use an uncertainty function $[eqn]$ defined over $[eqn]$ , and $[eqn]$ is defined over $[eqn]$ . The definition of the feasible set is the same as the one given in (62). In our channel model, this feasible set depends on the transition mapping N through (115) and (116).

We can now state the non-stochastic channel coding theorem for channels with transition mapping N.

Theorem 7.  For any totally bounded, normed metric space $[eqn]$ , channel with transition mapping N satisfying (115) and (116), and $[eqn]$ , we have

$$[eqn]$$

The proof is along the same lines as the one of Theorem 4 and is omitted.

Theorem 7 characterizes the capacity as the supremum of the mutual information over all codebooks in the feasible set. The following theorem shows that the same characterization is obtained if we optimize the right hand side in (117) over all codebooks in the space. It follows by Theorem 7 that rather than optimizing over all codebooks, a capacity-achieving codebook can be found within the smaller class $[eqn]$ of feasible sets with error at most $[eqn]$ .

Theorem 8.  The $[eqn]$ -capacity in (117) can also be written as

$$[eqn]$$

The proof is along the same lines as the one of Theorem 5 and is omitted.

6. Capacity of Stationary Memoryless Uncertain Channels

In this section, we consider the special case of stationary, memoryless, uncertain channels.

Let $[eqn]$ be the space of $[eqn]$ -valued discrete-time functions $[eqn]$ , where $[eqn]$ is the set of positive integers denoting the time step. Let $[eqn]$ denote the function $[eqn]$ restricted over the time interval $[eqn]$ . Let $[eqn]$ be a discrete set which represents a codebook. Also, let $[eqn]$ denote the set of all codewords up to time n and $[eqn]$ denote the set of all codeword symbols in the codebook at time n. The codeword symbols can be viewed as the coefficients representing a continuous signal in an infinite-dimensional space. For example, transmitting one symbol per time step can be viewed as transmitting a signal of unit spectral support over time. Any discrete-time function $[eqn]$ can be selected at the transmitter, sent over a channel, received with noise perturbation, and introduced by the channel. The perturbation of the signal at the receiver due to the noise can be described as a displacement experienced by the corresponding codeword symbols $[eqn]$ . To describe this perturbation, we consider the set-valued map $[eqn]$ , associating any point in $[eqn]$ to a set in $[eqn]$ , where $[eqn]$ is the space of $[eqn]$ -values discrete-time functions. For any transmitted codeword $[eqn]$ , the corresponding received codeword lies in the set

$$[eqn]$$

Also, the noise set associated with $[eqn]$ is

$$[eqn]$$

where $[eqn]$ We are now ready to define stationary, memoryless, uncertain channels.

Definition 14.  A stationary, memoryless, uncertain channel is a transition mapping $[eqn]$ that can be factorized into identical terms describing the noise experienced by the codeword symbols. Namely, there exists a set-valued map $[eqn]$ such that for all $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$

According to the definition, a stationary, memoryless, uncertain channel maps the nth input symbol into the nth output symbol in a way that does not depend on the symbols at other time steps, and the mapping is the same at all time steps. Since the channel can be characterized by the mapping N, to simplify the notation, we will use $[eqn]$ instead of $[eqn]$ .

Another important observation is that the $[eqn]$ -perturbation channel in Definition 9 may not admit a factorization like the one in (121). For example, consider the space to be equipped with the $[eqn]$ norm, the codeword symbols to represent the coefficients of an orthogonal representation of a transmitted signal, and the noise experienced by any codeword to be within a ball of radius $[eqn]$ . In this case, if a codeword symbol is perturbed by a value close to $[eqn]$ , the perturbation of all other symbols must be close to zero.

For stationary, memoryless, uncertain channels, all received codewords lie in the set $[eqn]$ , and the received codewords up to time n lie in the set $[eqn]$ . Then, for any $[eqn]$ , we let

$$[eqn]$$

where $[eqn]$ is an uncertainty function defined over the space of the received codewords. We also assume without loss of generality that at any time step n, the uncertainty associated with the space $[eqn]$ of received codewords is $[eqn]$ . We also let $[eqn]$ , where $[eqn]$ . Thus, $[eqn]$ is the set corresponding to the minimum uncertainty introduced by the noise mapping at a single time step. Finally, we let $[eqn]$ The quantity $[eqn]$ can be viewed as the confidence we have of not confusing $[eqn]$ and $[eqn]$ in any transmission or, equivalently, as the amount of adversarial effort required to induce a confusion between the two codewords. For example, if the uncertainty function is constructed using a measure, then all the erroneous codewords generated by an adversary to decode $[eqn]$ instead of $[eqn]$ must lie inside the equivocation set $[eqn]$ whose relative size is given by (122). The smaller the equivocation set is, the larger the effort required by the adversary to induce an error must be. If the uncertainty function represents the diameter of the set, then all the erroneous codewords generated by an adversary to decode $[eqn]$ instead of $[eqn]$ will be close to each other, in the sense of (122).

We now introduce the notion of a distinguishable codebook, ensuring that every codeword cannot be confused with any other codeword, rather than with a specific one, at a given level of confidence.

Definition 15. 

• $[eqn]$ -distinguishable codebook.*

For all $[eqn]$ and $[eqn]$ , a codebook $[eqn]$ is $[eqn]$ -distinguishable if for all $[eqn]$ , we have

$$[eqn]$$

It immediately follows that for any $[eqn]$ -distinguishable codebook $[eqn]$ , we have

$$[eqn]$$

so that each codeword in $[eqn]$ can be decoded correctly with confidence at least $[eqn]$ . Definion 15 guarantees even more, namely that the confidence of not confusing any pair of codewords is at least $[eqn]$ .

We now associate with any sequence $[eqn]$ the largest distinguishable rate sequence $[eqn]$ , whose elements represent the largest rates that satisfy that confidence sequence.

Definition 16.  Largest $[eqn]$ -distinguishable rate sequence.

For any sequence $[eqn]$ , the largest $[eqn]$ -distinguishable rate sequence $[eqn]$ is such that for all $[eqn]$ , we have

$$[eqn]$$

*where $[eqn]$ *

We say that any constant rate R that lays below the largest $[eqn]$ -distinguishable rate sequence is $[eqn]$ -distinguishable. Such a $[eqn]$ -distinguishable rate ensures the existence of a sequence of distinguishable codes that, for all $[eqn]$ , have a rate of at least R and confidence of at least $[eqn]$ .

Definition 17. 

• $[eqn]$ -distinguishable rate.*

For any sequence $[eqn]$ , a constant rate R is said to be $[eqn]$ -distinguishable if for all $[eqn]$ , we have

$$[eqn]$$

We now give our first definition of capacity for stationary, memoryless, uncertain channels as the supremum of the $[eqn]$ -distinguishable rates. Using this definition, transmitting at a constant rate below capacity ensures the existence of a sequence of codes that, for all $[eqn]$ , have confidence of at least $[eqn]$ .

Definition 18. 

• $[eqn]$ capacity.*

For any stationary, memoryless, uncertain channel with transition mapping N, and any given sequence $[eqn]$ , we let

$$[eqn]$$ $$[eqn]$$

Another definition of capacity arises if, rather than the largest lower bound to the sequence of rates, one considers the least upper bound for which we can transmit, satisfying a given confidence sequence. Using this definition, transmitting at a constant rate below capacity ensures the existence of a finite-length code (rather than a sequence of codes) that satisfies at least one confidence value along the sequence $[eqn]$ .

Definition 19. 

• $[eqn]$ capacity.*

For any stationary, memoryless, uncertain channel with transition mapping N, and any given sequence $[eqn]$ , we define

$$[eqn]$$

Next, consider Definition 19 in the case of $[eqn]$ as a constant sequence; namely, for all $[eqn]$ , we have $[eqn]$ . In this case, transmitting below capacity ensures the existence of a finite-length code that has confidence of at least $[eqn]$ . This is a generalization of the zero-error capacity,

Definition 20. 

• $[eqn]$ capacity.*

For any stationary, memoryless, uncertain channel with transition mapping N and any sequence $[eqn]$ , where for all $[eqn]$ we have $[eqn]$ , we define

$$[eqn]$$

Letting $[eqn]$ , we obtain the zero-error capacity. In this case, below capacity, there exists a code with which we can transmit with full confidence.

Finally, to give a definition of a non-stochastic analog of Shannon’s probabilistic capacity, we first say that any constant rate R is achievable if there exists a sequence $[eqn]$ as $[eqn]$ such that R lays below $[eqn]$ . An achievable rate R then ensures that for all $[eqn]$ , there exists an infinite sequence of distinguishable codes of rate of at least $[eqn]$ whose confidence tends towards one as $[eqn]$ . It follows that in this case, we can achieve communication at rate R with arbitrarily high confidence by choosing a sufficiently large codebook.

Definition 21.  Achievable rate.

A constant rate R is achievable if there exists a sequence $[eqn]$ such that $[eqn]$ as $[eqn]$ and

$$[eqn]$$

We now introduce the non-stochastic analog of Shannon’s probabilistic capacity as the supremum of the achievable rates. This means that we can pick any confidence sequence such that $[eqn]$ tends towards zero as $[eqn]$ . In this way, $[eqn]$ plays the role of the probability of error and the capacity is the largest rate that can be achieved by a sequence of codebooks with an arbitrarily high confidence level. Using this definition, transmitting at a rate below capacity ensures the existence of a sequence of codes achieving arbitrarily high confidence by increasing the codeword size.

Definition 22. 

• $[eqn]$ capacity.*

For any stationary, memoryless, uncertain channel with transition mapping N, we define the $[eqn]$ capacity as

$$[eqn]$$ $$[eqn]$$

We point out the key difference between Definitions 20 and 22. Transmitting below the $[eqn]$ capacity ensures the existence of a fixed codebook that has confidence of at least $[eqn]$ . In contrast, transmitting below the $[eqn]$ capacity allows us to achieve arbitrarily high confidence by increasing the codeword size.

To give a visual illustration of the different definitions of capacity, we refer to Figure 8.

For a given sequence $[eqn]$ , the figure sketches the largest $[eqn]$ -distinguishable rate sequence $[eqn]$ . According to definitions 18 and 19, the capacities $[eqn]$ and $[eqn]$ are given by the supremum and infimum of this sequence, respectively. On the other hand, according to Definition 22, the capacity $[eqn]$ is the largest limsup over all vanishing sequences $[eqn]$ . Assuming the figure refers to a vanishing sequence $[eqn]$ that achieves the supremum in (133), we have

$$[eqn]$$

We now relate our notions of capacity to the mutual information rate between transmitted and received codewords. Let X be the UV corresponding to the transmitted codeword. This is a map of $[eqn]$ and $[eqn]$ . Restricting this map to a finite time $[eqn]$ yields another UV $[eqn]$ and $[eqn]$ . Likewise, a codebook segment is a UV $[eqn]$ of marginal range $[eqn]$ Likewise, let Y be the UV corresponding to the received codeword. It is a map of $[eqn]$ and $[eqn]$ . $[eqn]$ and $[eqn]$ are UVs, and $[eqn]$ and $[eqn]$ . For a stationary, memoryless, uncertain channel with transition mapping N, these UVs are such that for all $[eqn]$ , $[eqn]$ and $[eqn]$ , and we have

$$[eqn]$$ $$[eqn]$$

Now, we define the largest $[eqn]$ -mutual information rate as the supremum mutual information per unit-symbol transmission that a codeword $[eqn]$ can provide about $[eqn]$ with confidence of at least $[eqn]$ .

Definition 23.  Largest $[eqn]$ -information rate.

For all $[eqn]$ , the largest $[eqn]$ -information rate from $[eqn]$ to $[eqn]$ is

$$[eqn]$$

In the following theorem. we establish the relationship between $[eqn]$ and $[eqn]$ .

Theorem 9.  For any totally bounded, normed metric space $[eqn]$ , disrete-time space $[eqn]$ , stationary, memoryless, uncertain channel with transition mapping N satisfying (135) and (136), and sequence $[eqn]$ such that, for all $[eqn]$ , we have $[eqn]$ , we have

$$[eqn]$$

We also have

$$[eqn]$$

Proof. The proof of the theorem is similar to the one of Theorem 4 and is given in Appendix B. □

The following coding theorem is now an immediate consequence of Theorem 9 and of our capacity definitions.

Theorem 10.  For any totally bounded, normed metric space $[eqn]$ , disrete-time space $[eqn]$ , stationary, memoryless, uncertain channel with transition mapping N satisfying (135) and (136), and sequence $[eqn]$ such that for all $[eqn]$ , $[eqn]$ and $[eqn]$ , we have

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

Theorem 10 provides multi-letter expressions of capacity, since $[eqn]$ depends on $[eqn]$ according to (137). In the Supplementary Materials, we establish some special cases of uncertainty functions, confidence sequences, and classes of stationary, memoryless, uncertain channels, leading to the factorization of the mutual information and to single-letter expressions.

7. Conclusions and Future Directions

We presented a non-stochastic notion of information with worst-case confidence and related it to the capacity of a communication channel subject to unknown noise. Using the non-stochastic variables framework of Nair [5] and a generalization of the Kolmogorov capacity allowing some amount of overlap in the packing sets [7], we showed that the capacity equals the largest amount of information conveyed by the transmitter to the receiver, with a given level of confidence. These results are the natural generalization of Nair’s ones, obtained in a zero-error framework, and provide an information-theoretic interpretation of the geometric problem of sphere packing with overlap, as studied in [7].

Non-stochastic approaches to information and their use to quantify the performance of various engineering systems have recently received attention in the context of estimation, control, security, communication over non-linear optical channels, and learning systems [8,9,10,11,12,13,14]. We hope that the theory developed here can be useful in the future in some of these contexts. While refinements and extensions of the theory are certainly of interest, explorations of application domains are of paramount importance. There is evidence in the literature regarding the need for a non-stochastic approach to study the flow of information in complex systems, and there is a certain tradition in computer science, especially in the field of online learning, to study various problems in both a stochastic and a non-stochastic setting [15,16,17]. Nevertheless, it seems that only a few isolated efforts have been made towards the formal development of a non-stochastic information theory. Wider involvement of the community in developing alternative, even competing, theories is certainly advisable to eventually fulfill the need of these application areas.