An algebraic framework for information theory: Classical Information

TL;DR

This paper introduces a comprehensive algebraic model for classical information theory, providing new computational tools and reformulating key theorems within an algebraic framework.

Contribution

It develops a complete algebraic framework for classical information theory, including probabilistic concepts and theorems, enhancing both theoretical understanding and computational methods.

Findings

Algebraic model successfully represents probabilistic concepts.

New computational tools derived from the algebraic framework.

Classical theorems reformulated and proved algebraically.

Abstract

This work proposes a complete algebraic model for classical information theory. As a precursor the essential probabilistic concepts have been defined and analyzed in the algebraic setting. Examples from probability and information theory demonstrate that in addition to theoretical insights provided by the algebraic model one obtains new computational and anlytical tools. Several important theorems of classical probahility and information theory are formulated and proved in the algebraic framework.

Equations

$$\mu (\bigcup _n{A}_n)=\sum _{n=1}^{\infty }\mu (A_n).$$

$$\begin{array}{rl}||x||& \ge 0\text{ and ∣∣a∣∣=0, iff a=0}\\ ||x+y||& \le ||x||+||y||\text{ (triangle inequality)}\\ ||xy||& \le ||x||||y||\text{ (Banach property) }\end{array}$$

$$||x{x}^{\ast }||=||x|{|}^2\text{ and }(xy{)}^{\ast }=y^{\ast }{x}^{\ast }\forall x,y\in B$$

$$r(x)=\underset{n\to \infty }{\lim }||x^n|{|}^{1/n}.$$

$$\begin{array}{rl}(f+g)(x)& =f(x)+g(x),(fg)(x)=f(x)g(x)\text{ and }\\ ||f||& =\underset{x\in X}{\sup }|f(x)|\forall f,g\in C(X)\end{array}$$

$$x_i^{\ast }=x_i,x_i{x}_j={\delta }_{ij}{x}_i,||x_i||=1\text{ and }\sum _i{x}_i=\text{mathbbm}1,$$

$$A=\bigcup _{k=0}^{\infty }{A}_k\text{ and }{A}_k\subset A_{k+1}\forall k$$

$$\begin{array}{rl}(cx)\otimes y& =x\otimes (cy)=c(x\otimes y)\\ (x+u)\otimes y& =x\otimes u+u\otimes y\text{ and }x\otimes (y+z)=x\otimes y+x\otimes z.\end{array}$$

$$\sum _i{a}_i(x_i\otimes y_i)\sum _j{b}_j(u_j\otimes z_j)=\sum _{ij}{a}_i{b}_j(x_i{u}_j\otimes y_i{z}_j)$$

$$\begin{array}{rl}{z}_{1_1}\otimes \cdots \otimes (a{z}_{i_k}+b{z}_{i_k}^{\prime })\otimes z_{i_{k+1}}\otimes \cdots =& a(z_{1_1}\otimes \cdots \otimes z_{i_k}\otimes z_{i_{k+1}}\otimes \cdots )+\\ & b(z_{1_1}\otimes \cdots \otimes z_{i_k}^{\prime }\otimes z_{i_{k+1}}\otimes \cdots ).\end{array}$$

$$(z_{i_1}\otimes z_{i_2}\otimes \cdots )(z_{i_1}^{\prime }\otimes z_{i_2}^{\prime }\otimes \cdots )=(z_{i_1}{z}_{i_1}^{\prime }\otimes z_{i_2}{z}_{i_2}^{\prime }\otimes \cdots )$$

$$||\sum _{i_1,i_2,\dots }{a}_{i_1{i}_2\cdots }{z}_{i_1}\otimes z_{i_2}\otimes \cdots ||=\sup \{|a_{i_1{i}_2\cdots }|\}$$

$${\left(\sum _{i_1,i_2,\dots }{a}_{i_1{i}_2\cdots }{z}_{i_1}\otimes z_{i_2}\otimes \cdots \right)}^{\ast }=\sum _{i_1,i_2,\dots }\overline{a_{i_1{i}_2\cdots }}{z}_{i_1}\otimes z_{i_2}\otimes \cdots$$

$$\mathcal{J}:\stackrel{\infty }{⨂}G\to \stackrel{\infty }{⨂}A\text{ and }{\mathcal{J}}^{\prime }:\stackrel{\infty }{⨂}A\to \stackrel{\infty }{⨂}G$$

$$\begin{array}{rl}& j(z_1\otimes \cdots \otimes z_k\otimes \text{mathbbm}1\otimes \cdots )=\varphi (z_1)\otimes \cdots \otimes \varphi (z_k)\otimes \text{mathbbm}1\otimes \cdots \text{ where }\\ & z_i\in \{y_0,y_1\}\text{ and }\varphi (y_0)=x_0,\varphi (y_1)=x_1\end{array}$$

$$\psi (x_r)=y_{b_0^r}\otimes y_{b_1^r}\otimes \cdots \otimes y_{b_k^r}$$

$$j^{\prime }(z_1\otimes \cdots \otimes z_k\otimes \text{mathbbm}1\otimes \cdots )=\varphi (z_1)\otimes \cdots \otimes \varphi (z_k)\otimes \text{mathbbm}1\otimes \cdots$$

$$\Omega ={\omega }_1\otimes {\omega }_2\otimes \cdots \text{ such that }{\omega }_i\in \mathcal{S}(A_i)$$

$$\Omega (\alpha )={\omega }_1(z_1){\omega }_2(z_2)\cdots {\omega }_k(z_k),$$

$${\int }_M{I}_B\mathrm{d}P=P(B)$$

$$x=a_1{P}_1+a_2{P}_2+\cdots +a_k{P}_k\text{ and }y=b_1{Q}_1+b_2{Q}_2+\cdots +b_l{Q}_l$$

$$\omega (f_i(x)g_j(y))=\omega (P_i{Q}_j)=\omega (P_i)\omega (Q_j).$$

$$\omega (x_1\cdots x_k)=\omega (x_1)\cdots \omega (x_k)$$

$$F_S(\mathtt{t})=\underset{\lambda }{\lim }\omega (e_{\lambda })$$

$$\underset{t_1,\dots ,t_n\to \infty }{\lim }F(t_1,\dots ,t_n)=1\text{ and }\underset{t_1,\dots ,t_n\to -\infty }{\lim }F(t_1,\dots ,t_n)=0$$

$$F(t_1,\dots ,t_n)=F(t_1)F(t_2)\cdots F(t_n).$$

$$e_m(\mathtt{t}+1/m)=m\chi (\mathtt{t}+1/m)(1+m\chi (\mathtt{t}+1/m){)}^{-1}\equiv \frac{m\chi (\mathtt{t}+1/m)}{1+m\chi (\mathtt{t}+1/m)}$$

$$\underset{\lambda }{\lim }\omega (e_{\lambda }(\mathtt{t}))=\underset{m\to \infty }{\lim }\omega (e_m(\mathtt{t}))$$

$$x=\sum a_i{P}_i\text{ such that }{P}_i{P}_j={\delta }_{ij}{P}_j\text{ and }{a}_i\ne 0\text{ distinct }$$

$${\mathcal{I}}_{\omega }(x)=\sum _i\omega (P_i)P_i$$

Full text

1 Introduction

The present paper proposes an algebraic model of classical information theory. We then carry out a detailed investigation of the model. The connection between operator algebras and information theory—both classical and quantum—have appeared in the scientific literature since the beginnings of information theory and operator algebras—both classical and quantum (see e.g. [Ume62, Seg60, Lin74, Ara75, Key02, BKK07, KW06]). The standard formulation of classical information theory [Ash90, CT99] on the other hand is sometimes seen as an important application of probability theory. Thus probabilistic concepts like distribution function, conditional expectation and independence are vital for the development of information theory. Most previous work including those mentioned above focus on some aspects of information theory, especially the noncommutative generalizations of the concepts of entropy and for specific probabilistic concepts they often resort to a representation on some Hilbert space. As a consequence, there does not appear to be a unified coherent approach based on intrinsically algebraic notions. The construction of such a model is one of the goals of the paper. As probabilistic concepts play such an important role in the development of information theory we devote a fairly large section to an algebraic approach to probability. It was I. E. Segal [Seg54], one of the major players in the early development of operator theory who first proposed such an algebraic approach to probability theory. Although we have mostly restricted ourselves to the discrete case, sufficient for our models of communication and information processes, our proposed model is different from Segal’s. We believe several aspects of our approach are novel (see the section-wise synopsis below) and yield deeper insights to information processes.

A strong motivation for this paper is the relatively young field of quantum information theory. It is almost folklore that in quantum mechanics we are forced to deal with noncommutative entities. Thus, the language of $C^{*}$ algebras, already known to physicists for decades [Haa92, Emc84] as “the algebra of observables” on which many extensions of classical probabilistic concepts can be made, became a natural setting for quantum information. As a complex quantum information scheme or protocol has several classical components (e.g. classical communication, coin-tosses etc.) it is important that we have a unified model and a single language for quantum and classical information. Such a formulation will be of great help in the difficult task of protocol analysis. Besides a unified framework will be of significant advantage for theoretical analysis. For example, a deeper study of quantum phenomena like (no) quantum broadcasting [BBLW07], quantum Huffman coding [BFGL00], channel capacity [Sch96] to name a few would benefit from the investigations of these structures. In this framework we may view a classical process as a special type of process described by commuting elements. Therefore, it seems appropriate to investigate this special case first. As we will show the classical structure is quite rich and sheds new light on some familiar aspects of information theory. There is yet another reason. In quantum mechanics we have several examples of observables taking only a finite number of values (the spectrum is finite). But in classical mechanics all variables take on a continuum of values. Therefore, we often see statements like “a finite-dimensional operator like spin is a purely quantum phenomenon that has no classical analogue”. However, when we talk about information systems finite-dimensional quantum systems have obvious classical analogues. A 2-dimensional quantum “source” corresponds to a classical binary source. Our investigations raise some questions about the possibility of an alternative formulation of probability theory with a more algebraic flavour [Seg54]. This is interesting in itself. But it is a side issue in this paper and will only be briefly commented upon. Since our main concern is the mathematical models of information processing systems we will be primarily dealing with discrete systems, thus circumventing some tricky topological issues.

Let us recall a simple model of a communication system proposed by Shanon [Sha48, SW49].

This model has essentially four components: source, channel, encoder/decoder and receiver. The source could be representing very different kinds of objects: a human speaker, a radar antenna or a distant star. We usually have some model of the source. The coding/decoding operation is required for three basic reasons: i) the source/receiver alphabet and the channel alphabet may be different, ii) to maximize the rate of information communication and ii) to detect and correct errors due to noise and distortion. Some amount of noise affects every stage of the operation. So the behavior of components are generally modeled as stochastic processes. This is valid in both the classical and often quantum communication processes. The difference, of course, is in the description of the two processes. As in any stochastic process, we specify the source by a family $X_{t}$ of random variables and the various stages of the communication system are modeled as (stochastic) transformations of these variables. The parameter $t$ can be continuous or discrete. In this work our primary focus will be on discrete processes corresponding to discrete time. Thus, a discrete source can be viewed as a generator of a countable set of random variables. Let us suppose the source “tosses” a coin and sends a 1 if it is “heads” and a 0 otherwise. We may model this by a pair of random variables $\{X_{h},X_{T}\}$ on the probability space $\{h\text{ (heads)},t\text{ (tails)}\}$ such that $X_{h}(h)=X_{t}(t)=1\text{ and }X_{h}(t)=X_{t}(h)=0$. If the coin is unbiased we say that the state of the source is given by a probability measure $\{1/2,1/2\}$. In general it is $\{p,q\}$ where $0\leq p=1-q\leq 1$ is the probability of heads. This simple model can be generalized to more complicated sources. Besides these elementary random variables we encounter functions of these variables. Thus we are led to study algebras of random variables. Recall the standard definition of a random variable: it is a (measurable) function on a probability space $S$. Hence, in the standard formulation we need a probability space or sample space to define our random variables or “observables”. Let us also recall that a probability space is a triple $(S,{\mathcal{M}},\mu)$, where $S$ is a set, the set of elementary or atomic events, ${\mathcal{M}}$ is a $\sigma$-algebra of subsets and $\mu$ is the probability measure. Thus if $\{A_{n}\}$ is a sequence of mutually disjoint elements from ${\mathcal{M}}$ then

[TABLE]

Moreover, $\mu(S)=1$ and $\mu(B)\geq 0$ for any $B\in{\mathcal{M}}$. These are essentially the Kolmogorov axioms. A real or complex valued random variable is a measurable function form $S\rightarrow\mathbb{R}$ or $S\rightarrow\mathbb{C}$. Here measurability is with respect to the Borel $\sigma$-algebra of $\mathbb{R}$ or $\mathbb{C}$. We recall that the Borel $\sigma$-algebra of any topological space is generated by its open sets. So in some sense in this formulation the probability space is fundamental and the notion of random variables is based on the former. However, from an observer/experimenter point of view the random variables are the basic entities because these are precisely the observables. In statistical theories like information theory it is the set of random variables and their distributions and transformations which are of primary interest. Of course, to compute the probability distributions of the random variables we have to appeal to the original probability space. But once the distributions have been determined for almost all computations they suffice and the underlying probability or sample space plays little role. The fundamental theorem of Kolmogorov [Bil95, Shi84] guarantees that given a set of random variables and their distributions satisfying certain consistency conditions we can reconstruct a probability space giving these distributions. These observations suggest that we take the algebra of random variables or observables as our primary structure and derive all relevant quantities from this structure. One of the advantages is that we deal with a smaller spaces restricted to quantities of interest. In the modeling of security protocols this a more realistic approach since different participants have access to different sets of observables and may assign different probability structures on the same set of events.

In the quantum case there are more fundamental reasons for working with the algebras of observables. We will not go into these here. The current work is an attempt at formulating (classical) information theory in an algebraic framework. We will mainly focus on $C^{*}$ and von Neumann algebras. We will see that most interesting spaces of observables do have a $C^{*}$ structure. As mentioned before, we will be dealing with discrete spaces in this work. We also observe that $C^{*}$ algebras have been studied intensively since the pioneering works of Murray, von Neumann, Gelfand, Naimark and Segal and others starting from 1930s. As we stated at the beginning of this section, several probabilistic and information theoretic concepts like conditional expectation, entropy, differential entropy have previously been investigated in the algebraic context. However to the best of our knowledge there is no work investigating information and communication theory in a purely algebraic framework. Our investigations indicate that most if not all important concepts and constructs of information theory can be dealt with in the algebraic framework. The paper is structured as follows.

In Section 2 we give the basic definitions of the algebras of interest. This section is fairly detailed as we provide proofs of several structure theorems for finite-dimensional abelian $C^{*}$ algebras and their tensor products, possibly infinite. There are two reasons for this. The first is to make the paper as self-contained as possible. The second reason is to demonstrate the power and utility of the algebraic techniques. Moreover, we believe that in these special cases some of the proofs are new. We also give several examples.

Section 3 gives an account of probabilistic concepts from an algebraic perspective. In particular, we investigate the fundamental notion of independence and demonstrate how it relates to the algebraic structure. We note that there is a very sophisticated theory of noncommutative or “free probability”[VDN92]. Our approach in the simpler commutative case is different in several aspects. One important point in which our approach seems novel is the definition of a probability distribution function. The definition we give is algebraic in the sense that it depends on the intrinsic properties of the algebra. Specifically, we define a probability distribution function as the weak limit of a net or sequence of elements in a subalgebra representing an approximate identity of an ideal or a subalgebra. To illustrate the practical use of these techniques we give some typical examples from standard probability theory. The problem of “waiting time” shows that the algebraic approach can offer new techniques and insights. Finally, using the definition of distribution function and some other constructs we formulate and prove some of the basic limit theorems in this framework. These are used later in proving results in information theory.

In Section 4 we give a precise algebraic model of information communication system. The fundamental concept of entropy is introduced as a limiting value of typical sequences of the algebra. The notion of typical sequence comes from the limit theorems. In the conventional approach the limit is taken in the probability (convergence in measure). In our algebraic case it corresponds to a weak limit. The point is, we can do all this in purely algebraic setting. We also define and study the crucial notion of a channel. In particular, the channel coding theorem is presented as an approximation result. Stated informally,

Every channel other than the useless ones can be approximated by a lossless channel under appropriate coding.

In the final section we summarize our constructions and discuss future work.

2 Algebraic Preliminaries

An algebra ${\mathcal{A}}$ is vector space over a field $\tt{F}$ with an associative bilinear product: ${\mathcal{A}}\times{\mathcal{A}}\rightarrow{\mathcal{A}}$. We take ${\tt F}=\mathbb{C}$, the field of complex numbers. We deal mostly with unital algebras, that is, algebras with a unit $\mathbbm{1}$. A Banach algebra is an algebra with a non-negative real function $|\!||\!|$ on ${\mathcal{A}}$ such that

[TABLE]

and ${\mathcal{A}}$ is complete in the topology defined by the norm. A $C^{*}$ algebra $B$ is a Banach algebra with an anti-linear involution ∗ (a map $\sigma$ is an involution if $\sigma^{2}=1$, it is antilinear if $\sigma(x+cy)=\sigma(x)+\overline{c}\sigma(y),\;c\text{ a complex number}$) such that

[TABLE]

This implies that $|\!|x|\!|=|\!|x^{*}|\!|$. The quintessential examples of a $C^{*}$ algebra are the norm-closed subalgebras of ${\mathcal{L}}(H)$, the set of bounded operators on a Hilbert space of $H$. The fundamental Gelfand-Naimark-Segal (GNS) theorem states that every $C^{*}$ algebra can be isometrically embedded in some ${\mathcal{L}}(H)$. The notion of the spectrum of an operator has an algebraic analogue without reference to the representation space. The resolvent of an element $x$ in the $C^{*}$ algebra $B$ is the set $R(x)\subset\mathbb{C}$ such that $\lambda\in R(x)$ implies $\lambda\mathbbm{1}-x$ is invertible. The spectrum ${\tt sp}(x)$ is the complement of the resolvent. The spectrum is a nonempty closed and bounded subset and hence compact. Define $r(x)=\sup\{|\lambda|:\lambda\in{\tt sp}(x)\}$, the spectral radius. A basic result states that

[TABLE]

An element $x$ is self-adjoint if $x=x^{*}$, normal if $x^{*}x=xx^{*}$ and positive (strictly positive) if $x$ is self-adjoint and ${\tt sp}(x)\subset[0,\infty)((0,\infty))$. A self-adjoint element has a real spectrum and conversely. Since $x=(x+x^{*})/2+i(x-x^{*})/2i$ any element of a $C^{*}$ algebra can be decomposed into self-adjoint “real” ($(x+x^{*})/2$) and “imaginary” ($(x-x^{*})/2i$) parts. For a self-adjoint element $x$, $r(x)=|\!|x|\!|$. Thus a positive element is self-adjoint. The positive elements define a partial order on $A$. Thus $x\leq y$ iff $y-x\geq 0$ (positive). An important property of positive elements is that they have unique square-roots. Thus if $a\geq 0$ there is a unique element $b\geq 0$ such that $b^{2}=a$. We write $\sqrt{a}\text{ or }a^{1/2}$ for the square-root. Since $x^{*}x\geq 0$ it has a unique square-root. If $x$ is normal we write $|x|=\sqrt{x^{*}x}$. In particular, if $x$ is self-adjoint, $|x|=\sqrt{x^{2}}$. A self-adjoint element $x$ has a decomposition $x=x_{+}-x_{-}$ into positive and negative parts where $x_{+}=(|x|+x)/2\ \text{ and }x_{-}=(|x|-x)/2)$ are positive. An element $p\in B$ is a projection if $p$ is self-adjoint and $p^{2}=p$. Given two $C^{*}$-algebras $A$ and $B$ a homomorphism $F$ is a linear map preserving the product and ∗ structures. It is continuous iff bounded. A continuous isomorphism of $C^{*}$ algebras is an isometry (norm preserving). A homomorphism is positive if it maps positive elements to positive elements. A (linear) functional on $A$ is a linear map $A\rightarrow\mathbb{C}$. The GNS construction starts with a positive functional (mapping positive elements to non-negative numbers) on $B$. The details may be found in [KR97, Tak02]. A positive functional $\omega$ such that $\omega(\mathbbm{1})=1$ is called a state. The set of states $G$ is convex. The extreme points are called pure states and $G$ is the convex closure of pure states (Krein-Millman theorem). A set $B\subset A$ is called a subalgebra if it is a $C^{*}$ algebra with the inherited product. That is, it is a subalgebra in the algebraic sense and it is closed in the norm topology. A subalgebra is $B$ called unital if it contains the identity of $A$. Our primary interest will be on abelian (also called commutative) algebras. The structure theory is a bit different in this case. Of course, the GNS construction is valid and the elements of the algebra act as multiplication operators on the representing Hilbert space. However, there is an alternative representation in the abelian case due to Gelfand and Naimark which will be of primary interest to us. To motivate it consider an example.

Let $X$ be a compact Hausdorff topological space, for example, a closed and bounded set in $\mathbb{R}^{n}$. Let $C(X)$ denote the space continuous complex functions on $X$. It includes the constant functions. If we define addition and multiplication point-wise

[TABLE]

then $C(X)$ becomes a complex Banach algebra. If we define $f^{*}(x)=\overline{f(x)}$ then $C(X)$ is an abelian $C^{*}$ algebra. This is a prototype of abelian $C^{*}$ algebras [KR97]. One can generalize to (essentially) bounded measurable functions on measure spaces with appropriate norm. However, for the purposes of this paper it suffices to consider compact spaces with measures defined on Borel $\sigma$-algebras. We will dwell more on this point in the next section. A complex function (not necessarily continuous) is called simple if its range is finite. For example, the indicator function $I_{S}$ of a subset $S\subset X$, given by $I_{S}(x)=1\text{ if }x\in S$ and 0 otherwise is a simple function. It is not continuous unless $S=X$ or $S$ is a connected component. Simple functions play a crucial role in probability and integration theory. From their definition it follows that the projections in $C(X)$ are precisely the indicator functions. The constant functions 1 and 0 are both projections corresponding to $S=X\text{ and }\emptyset$ resp. These are the only projections in $C(X)$ if $X$ is connected. The basic structure theorem for abelian $C^{*}$ algebras is the following.

Theorem 1**.**

An abelian $C^{*}$ algebra with unity is isomorphic to the algebra $C(X)$ for a compact Hasudorff space $X$. The isomorphism is an isometry (norm preserving).

The main idea of the proof comes from the following observation. In any function algebra $C(X)$ for $p\in X$ the map $\sigma_{P}:f\rightarrow f(p),\;f\in C(X)$ is a linear functional on $C(X)$. These are multiplicative functionals in the sense that $\sigma_{p}(xy)=\sigma_{p}(x)\sigma_{p}(y)$. In fact these are the only possible multiplicative functionals. The Gelfand representation for an abstract abelian $C^{*}$ algebra $A$ identifies the space $X$ as the set of multiplicative functionals and gives it a topology to make these continuous. The details can be found in [KR97].

Now let $X=\{a_{1},\dotsc,a_{n}\}$ be a finite set with discreet topology. Then $A=C(X)$ is the set of all functions $X\rightarrow\mathbb{C}$. The algebra $C(X)$ can be considered as the algebra of (complex) random variables on the finite probability space $X$. Let $x_{i}(a_{j})=\delta_{ij},\;i,j=1,\dotsc,n$. Here $\delta_{ij}=1\text{ if }i=j\text{ and }0$ otherwise. The functions $x_{i}\in A$ form a basis for $A$. Their multiplication table is particularly simple: $x_{i}x_{j}=\delta_{ij}x_{i}$. They also satisfy $\sum_{i}x_{i}=\mathbbm{1}$. These are projections in $A$. They are orthogonal in the sense that $x_{i}x_{i}=0\text{ for }i\neq j$. We call any basis consisting of elements of norm 1 with distinct elements orthogonal atomic. A set of linearly independent elements $\{y_{i}\}$ satisfying $\sum_{i}y_{i}=\mathbbm{1}$ is said to be complete. The next theorem gives us the general structure of any finite-dimensional algebra.

Theorem 2**.**

Let $A$ be a finite-dimensional abelian $C^{*}$ algebra. Then there is a unique (up to permutations) complete atomic basis ${\mathcal{B}}=\{x_{1},\dotsc,x_{n}\}$. That is, the basis elements satisfy

[TABLE]

Let $x=\sum_{i}a_{i}x_{i}\in A$. Then ${\tt sp}(x)=\{a_{i}\}$ and hence $|\!|x|\!|=\max_{i}\{|a_{i}|\}$.

Proof.

Let $\{y_{1},\dotsc,y_{n}\}$ be a basis for $A$. Since the self-adjoint elements $(y_{i}+y_{i}^{*})/2$ and $i(y_{i}-y_{i}^{*})/2$ span $A$ we can choose an independent set. Hence, we may assume that the $y_{i}$ are self-adjoint. Then each $y_{i}^{2}$ is positive and hence possesses a square-root $|y_{i}|$. Moreover, $|y_{i}|\geq y_{i}$ [KR97, Bra02].222Bratteli (pp. 35) gives a proof which does not use Gelfand representation. We can therefore write each $y_{i}=(|y_{i}|+y_{i})/2-(|y_{i}|-y_{i})/2$, as the difference of two positive elements. Again, choosing an independent set we may assume that $y_{i}$ themselves are positive with norm 1. Let $S=\{z:z\geq 0\text{ and }|\!|z|\!|\leq 4$. $S$ is convex and compact (being closed and bounded) and $y_{i}\in S$. Hence, by the Krein-Millman theorem [KR97] $S$ is the convex closure of its extreme points333Recall that extreme points of a convex set are those which cannot be written as a non-trivial convex combination of some members of the set. We may assume that these extreme points have norm 1 (obviously discarding 0). Since each $y_{i}$ can be written as a finite convex sum of its extreme points we can pick a basis $x_{1},\dotsc,x_{n}$ of extreme points. We complete the proof by showing that the $x_{i}$’s satisfy equations (2) and that they are unique.

Now $|\!|x_{i}|\!|=1$ implies that for any $|\lambda|>1$, $\lambda-x_{i}=\lambda(\mathbbm{1}-\lambda^{-1}x_{i})$ is invertible. This can be proved by using the geometric series of $(1-\lambda^{-1}x_{i})^{-1}$. Hence if $a\in{\tt sp}(x_{i})$ then $0\leq a\leq 1$ and $\mathbbm{1}-x_{i}$ is positive. Since ${\tt sp}(x_{i}-x_{i}^{2})=\{a-a^{2}:a\in{\tt sp}(x_{i})\}$ and $a-a^{2}\geq 0$ it follows that $x_{i}-x_{i}^{2}\geq 0$. As $x_{i}=(2(x_{i}-x_{i}^{2})+2x_{i}^{2})/2$, a convex combination of two positive elements in $S$ and $x_{i}$ is a non-zero extreme point we must have $x_{i}-x_{i}^{2}=0$ or $x_{i}-x_{i}^{2}=x_{i}^{2}$. The last possibility is ruled out because it would imply $|\!|x_{i}|\!|=2|\!|x_{i}^{2}|\!|=2|\!|x_{i}|\!|^{2}=2$. Hence $x_{i}^{2}=x_{i}$. To prove that they are orthogonal observe that $x_{i}-x_{i}x_{j}=x_{i}(1-x_{j})$ is positive. Thus $x_{i}=(2x_{i}(1-x_{j})+2x_{i}x_{j})/2$ is a convex combination of points in $S$. Hence, as before either $x_{i}x_{j}=0$ or $x_{i}=x_{i}x_{j}$. With $x_{j}$ in place of $x_{i}$ we conclude that $x_{i}x_{j}=0$ or $x_{j}=x_{i}x_{j}$. Thus the only possibility for $x_{i}\neq x_{j}$ is that $x_{i}x_{j}=0$.

To prove the decomposition property let $\mathbbm{1}=\sum_{i}a_{i}x_{i}$. Squaring and using the orthogonality of $x_{i}$’s we conclude that $a_{i}=1\text{ or }0$. If some $a_{k}=0$ then $x_{k}=x_{k}\mathbbm{1}=x_{k}\sum_{i}a_{i}x_{i}=0$. Hence, all $a_{k}=1$. Finally, let $\{z_{i}\}$ be another basis satisfying (2). Let $z_{i}=\sum_{j}b_{ij}x_{j}$. As before, $b_{ij}=1\text{ or }0$ and the matrix $(b_{ij})$ is a 0-1 matrix. For fixed $i$ let $T_{i}$ be the set of integers $j$ such that $b_{ij}=1$. Then $x_{i}x_{j}=0,\;i\neq j$ implies $T_{i}$ and $T_{j}$ are disjoint. This along with the last condition in (2) implies that $T_{i}$’s form a partition of the set $\{1,\dotsc,n\}$. Thus each $T_{i}$ is a singleton and the matrix $(b_{ij})$ has exactly one 1 in each row and column. It is a permutation matrix.

Let $x=\sum_{i}a_{i}x_{i}$ be an element of $A$. Then $\lambda\mathbbm{1}-x=\sum_{i}(\lambda-a_{i})x_{i}$. This is invertible iff $\lambda\neq a_{i},\;i=1,\dotsc,n$ with inverse $\sum_{i}(\lambda-a_{i})^{-1}x_{i}$. The proof is complete. ∎

Let us observe that we could have proved the theorem using the Gelfand representation. But the above proof is more intrinsic depending mostly on the structure of the algebra itself only.

Corollary 1**.**

Let $A$ be an abelian $C^{*}$-algebra satisfying the following conditions. There are finite-dimensional subalgebras $A_{k},\;k=0,1,\dotsc$ with

[TABLE]

and for each $k$ corresponding to $A_{k}$ there is complementary subalgebra $A_{k}^{\prime}\subset A_{k+1}$ such that $A_{k}A_{k}^{\prime}=A_{k+1}$, $A_{k}\bigcap A_{k}^{\prime}=\{0,\mathbbm{1}\}$ and for $x\in A_{k},y\in A_{k}^{\prime}$ implies $xy\neq 0$ unless $x$ or $y$ is 0. Then there is a countable basis for $A$ satisfying the first three equations in (2).

Proof.

We prove by induction. The case of $A_{0}$ is proved in the theorem. Assume we have an atomic basis $\{y^{n}_{1},\dotsc,y^{n}_{k_{n}}\}$ for $A_{n}$. There is a (unique) atomic basis $\{x^{n}_{1}\,\dotsc,x^{n}_{m_{n}}\}$ in $A_{n}^{\prime}$. It is now a routine matter to show that $\{x^{n}_{i}y^{n}_{j}:1\leq k_{n}\text{ and }1\leq m_{n}\}$ form a basis in $A_{n+1}$. ∎

The conditions in the corollary can be slightly weakened by requiring that there be embeddings (injective algebra homomorphisms) $\alpha_{k}:A_{k}\rightarrow A_{k+1}$ and $\alpha^{\prime}_{k}:A^{\prime}_{k}\rightarrow A_{k+1}$ such that the images $\alpha_{k}(A_{k})$ and $\alpha^{\prime}_{k}(A^{\prime}_{k})$ satisfy the conditions stated. Such a structure will appear in the tensor product of algebras to be defined below. They play an important role in our modeling of information and communication systems. Let us also note that the basis structure in Theorem 2 may be used to defined a finite dimensional $C^{*}$ algebra abstractly.

2.1 Tensor products

We next describe an important construction for $C^{*}$ algebras. Given two $C^{*}$ algebras $A$ and $B$, the tensor product $A\otimes B$ is defined as follows. As a set it consists of all finite linear combinations of symbols of the form $\{x\otimes y:x\in A,y\in B\}$ subject to the conditions that for all $x,u\in A,\;y,z\in B\text{ and }c\in\mathbb{C}$,

[TABLE]

Thus the tensor product is bilinear. There are no other relations. Note that by definition the products of the form $x\otimes y$ span $A\otimes B$. Hence, if $\{x_{i}\}\text{ and }\{y_{j}\}$ are bases for $A$ and $B$ respectively then $\{x_{i}\otimes y_{j}\}$ is a basis for $A\otimes B$. The linear space $A\otimes B$ becomes an algebra by defining $(x\otimes y)(u\otimes z)=xu\otimes yz$ and extending by bilinearity. Explicitly,

[TABLE]

The $*$ is defined by $(x\otimes y)^{*}=x^{*}\otimes y^{*}$ and extending anti-linearly. The problem is defining the norm since it is not a linear function. In fact, for general $C^{*}$ algebras there could be a number of inequivalent norms on different completions of $A\otimes B$. This problem of non-uniqueness, however, does not exist if one of the factors is abelian or finite-dimensional. Since, in this work we will be primarily concerned with abelian algebras this point will not be discussed further. Our basic model will be an infinite tensor product of finite dimensional $C^{*}$ algebras which we present next.

Let $A_{k},\;k=1,2,\dotsc,$ be finite dimensional abelian $C^{*}$ algebras with atomic basis $B_{k}=\{x_{k1},\dotsc,x_{kn_{k}}\}$. Let $B^{\infty}$ be the set consisting of all infinite strings of the form $z_{i_{1}}\otimes z_{i_{2}}\otimes\cdots$ where all but a finite number ($>0$) of $z_{i_{k}}$s are equal to $\mathbbm{1}$ and if some $z_{i_{k}}\neq\mathbbm{1}$ then $z_{i_{k}}\in B_{k}$. Explicitly, $B^{\infty}$ consists of strings of the form $z_{i_{1}}\otimes z_{i_{2}}\otimes\cdots\otimes z_{i_{k}}\otimes\mathbbm{1}\otimes\mathbbm{1}\otimes\cdots,\;k=1,2,\dotsc$ and $z_{i}\in B$. Let $\tilde{\mathfrak{A}}=\otimes_{i=1}^{\infty}{A}_{i}$ be the vector space with basis $B^{\infty}$ such that $z_{i_{1}}\otimes z_{i_{2}}\otimes\cdots\otimes z_{i_{k}}\otimes\cdots$ is linear in each factor separately:

[TABLE]

Clearly every $\alpha\in\tilde{\mathfrak{A}}$ is a finite linear combination of elements in $B^{\infty}$. We define a product in $\tilde{\mathfrak{A}}$ as follows. First, for elements of $B^{\infty}$:

[TABLE]

We extend the product to whole of $\tilde{\mathfrak{A}}$ by linearity. Next define a norm by

[TABLE]

It is straightforward to show that $B^{\infty}$ is an atomic basis. It follows that the above function is indeed an algebra norm and that $\tilde{\mathfrak{A}}$ is an abelian normed algebra. We also define $*$-operation by

[TABLE]

It is routine to check that for $x\in\tilde{\mathfrak{A}}$, $|\!|xx^{*}|\!|=|\!|x|\!|^{2}$. Finally, we complete the norm and call the resulting $C^{*}$ algebra $\mathfrak{A}$. The completion of a norm is a technical device that uses the fact that any normed algebra $X$ can be isometrically mapped to a norm complete algebra (a Banach algebra) $\hat{X}$ and the image $X$ is dense in $\hat{X}$ (see [KR97]).444There are some delicate convergence issues here. Since $\tilde{\mathfrak{A}}$ consisting of finite sums of tensor products is dense in $\mathfrak{A}$ it often suffices to prove some statement about $\tilde{\mathfrak{A}}$ and extended it to $\mathfrak{A}$ by continuity. With these definitions $\mathfrak{A}$ is a $C^{*}$ algebra. An important special case is when all the factor algebras $A_{i}=A$. We then write the infinite tensor product $C^{*}$ algebra as $\bigotimes^{\infty}{A}$. Intuitively, the elements of an atomic basis $B^{\infty}$ of $\bigotimes^{\infty}{A}$ correspond to strings from an alphabet (represented by the basis $B$) with a given prefix. A general element of $A$ which is a linear combination of elements of $\bigotimes^{\infty}{B}$. Of particular interest is the 2-dimensional algebra $D$ corresponding to a binary alphabet. Thus we name $\bigotimes^{\infty}{D}$ the binary algebra. Let us fix some notation. For any finite dimensional $C^{*}$ algebra $A$ the atomic basis $B^{\infty}$ for $\bigotimes^{\infty}{A}$ constructed above will be denoted by $B^{\infty}_{A}$ to emphasize the association. The algebras $\bigotimes^{\infty}{A}$ will be our model of signals from a source/encoder which are strings (of arbitrary length) from some alphabet. We next prove a result that is relevant for coding theory.

Proposition 1**.**

Let $A$ be an abelian $C^{*}$ algebra of dimension $n$ with atomic basis $B_{A}=\{x_{0},\dotsc,x_{n-1}\}$. Let $B_{G}=\{y_{0},y_{1}\}$ be the atomic basis of the 2-dimensional algebra $G$ defined above. Then there are injective algebra homomorphisms

[TABLE]

that are isometries.

Proof.

We observe that it is sufficient to define an injective set map $j$ (resp. $j^{\prime}$) from $B_{G}^{\infty}$ to $B_{A}^{\infty}$ (resp. $B_{A}^{\infty}$ to $B_{G}^{\infty}$). For we can first extend these to linear maps ${\mathcal{J}}$ (resp. ${\mathcal{J}}^{\prime}$) on the appropriate spaces. The fact that the bases are atomic will ensure that these are injective algebra homomorphisms, in fact, isometries. Let

[TABLE]

To construct $j^{\prime}$ let the binary representation of the integer $n-1$ be of length $k+1$ where $k=\lfloor\log_{2}{n}\rfloor$. For $0\leq r\leq n-1$ $r=b^{r}_{0}+b^{r}_{1}2+b^{r}_{2}2^{2}+\cdots+b^{r}_{k}2^{k}$ be the binary representation of $r$ of length $k+1$ (pad it with 0’s if necessary). Let $\psi:B_{A}\rightarrow B^{\infty}_{G}$ be the map defined by

[TABLE]

extend it to a map $j^{\prime}:B^{\infty}_{A}\rightarrow B^{\infty}_{G}$ by

[TABLE]

The map $j^{\prime}$ is injective and the proof is complete. ∎

Let us note that from the injective maps $j\text{ and }j^{\prime}$ we can construct a bijective correspondence between $B^{\infty}_{A}\text{ and }B^{\infty}_{G}$ by a Schroeder-Bernstein type construction (see [Kle52]) and this can be lifted to an algebra isometry. But for us, the isomorphisms induced by maps like $j$ and $j^{\prime}$ (these are certainly not unique) will be greatest interest. Essentially, what the proposition says is that it is often sufficient to restrict our attention to the special algebra $\bigotimes^{\infty}{G}$.

The next step is to describe the state space. We recall that states of an algebra $A$ are precisely the positive functionals $\omega$ that are normalized: $\omega(\mathbbm{1})=1$. Given a $C^{*}$ subalgebra $V\subset A$ the set of states of $V$ will be denoted by $\mathscr{S}(V)$. Let $\mathfrak{A}=\otimes^{\infty}_{i=1}A_{i}$ denote the infinite tensor product of finite-dimensional algebras $A_{i}$. An infinite product state of $\mathfrak{A}$ is a functional of the form

[TABLE]

This is indeed a state of $\mathfrak{A}$ for if $\alpha_{k}=z_{1}\otimes z_{2}\otimes\cdots\otimes z_{k}\otimes\mathbbm{1}\otimes\mathbbm{1}\cdots\in\mathfrak{A}$ then

[TABLE]

a finite product. Since an arbitrary element of $\mathfrak{A}$ is the limit of sequence of finite sums of elements of the form $\alpha_{k},\;k=1,2,\dotsc$ $\Omega$ is bounded by the principle of uniform boundedness. Clearly, it is positive. A general state on $\mathfrak{A}$ is a convex combination of product states like $\Omega$.

2.2 Analytic functions on $C^{*}$ algebras

In this section we discuss another useful construction. Let $A$ be a $C^{*}$ algebra. Suppose $f(z)$ is an analytic function whose Taylor series $\sum_{n=0}^{\infty}a_{n}(z-c)^{n}$ is convergent in a region $|z-c|<R$. The convergence of the series $\sum|\!|x-c\mathbbm{1}|\!|^{n}$ for $|\!|x-c\mathbbm{1}|\!|<R$ implies that the series $\sum_{n=0}^{\infty}(x-c\mathbbm{1})^{n}$ converges (we need completeness of $A$ for this). Thus it makes sense to talk of analytic functions on a $C^{*}$ algebra. If we have an atomic basis $\{x_{1},x_{2},\dotsc\}$ in an abelian $C^{*}$ algebra then the functions are particularly simple in this basis. Thus if $x=\sum_{i}a_{i}x_{i}$ then $f(x)=\sum_{i}f(a_{i})x_{i}$ provided that $f(a_{i})$ are defined in an appropriate domain. We will mostly take this as our definition with the understanding that the constant function $c$ is identified with $c\mathbbm{1}$.

3 Algebraic approach to probability

We have observed that discrete signals from a source are modeled by an abelian algebra. The elements of the algebra correspond to random variables representing the output of the source. With random variables we always associate a probability distribution. In the standard treatment of probability theory the probability or sample space is introduced first. Random variables are defined as (measurable) real (or complex, in general) functions on this space. One then finds the probability distributions of the random variables and most important quantities like mean, variance and correlations are based on these distributions. In particular, the mean or expectation value plays a central role. Note that random variables can be added and multiplied making it a real algebra (scalars are the constant random variables). Note also that random variables also represent quantities that are actually measured or observed- the voltage across a resistor, the currents in an antenna, the position of a Brownian particle and so on. The probability distribution corresponds to the state of the devices that produce these outputs. We will take the alternative view and start with these observables as our basic objects. In this way, we single out the objects which are relevant to a specific problem. In the following paragraphs we formalize these notions.

3.1 Basic notions

A classical observable algebra is an abelian complex $C^{*}$ algebra $A$. It is convenient to use complex algebras. We can restrict our attention to real algebras whenever necessary. Recall that a state on $A$ is positive linear functional $\omega$ such that $\omega(\mathbbm{1})=1$. We can identify $\omega$ with a probability measure as follows. Suppose $(M,{\mathcal{S}},P)$ is probability space, ($M$= sample space, ${\mathcal{S}}$ = $\sigma$-algebra, $P$=probability measure). Let $L_{\infty}(M,{\mathcal{S}},P)$ (or simply $L_{\infty}(M)$ if the measure structure is clear) be the set of essentially bounded measurable complex functions.555A function $f$ is said to be essentially bounded if there is a constant $K$ such that $|f(x)|\leq K$ almost everywhere. The essential is the infimum over all such $K$: $\mathrm{ess}\;\sup(|f|)=\inf\{k:P\{x:|f(x)|>k\}=0\}$. We can give it a $C^{*}$ structure as in the case of $C(X)$, the space of continuous functions on a compact topological space $X$ (see equation (1)), but using the essential supremum instead of the ordinary supremum. If $B\in{\mathcal{S}}$ then the indicator function $I_{B}\in L^{\infty}(M,\mathbb{C})$ and

[TABLE]

where the integral is defined in the sense of Lebesgue. Note that $\omega_{P}(f)\equiv\int fdP$ is a positive linear functional on $L_{\infty}(M)$. Since $\omega_{P}(\mathbbm{1})=1$ it is a state.

Definition 1**.**

A probability algebra is a pair $(A,S)$ where $A$ is an observable algebra and $S\subset{\mathscr{S}}(A)$ is a set of states. A probability algebra is defined to be fixed if $S$ contains only one state. A probability algebra ${\mathscr{A}}_{1}=(A_{1},S_{1})$ is defined to be a cover of another ${\mathscr{A}}_{2}=(A_{2},S_{2})$ if there is an algebra homomorphism $\phi:A_{1}\rightarrow A_{2}$ and a one-to-one correspondence $\gamma:S_{1}\leftrightarrow S_{2}$ such that the following conditions hold: i. $\phi$ is onto and ii. for all $x\in A_{1}$ and $\omega\in S_{1}$: $\omega(x)=\gamma(\omega)(\phi(x))$.

Let $\omega$ be a state on an abelian $C^{*}$ algebra $A$. Call two elements $x,y\in A$ uncorrelated in the state $\omega$ if $\omega(xy)=\omega(x)\omega(y)$. Note that this definition depends crucially on the state: the same two elements can be correlated in some other state $\omega^{\prime}$. Two natural questions are immediate. Are there any states for which every pair of elements of $A$ are uncorrelated? Are there a pair of elements which are uncorrelated in every state? Two trivial candidates for the second question are $\mathbbm{1}$ and 0. Either of them is uncorrelated to every element. We implicitly exclude these two trivial cases. Concerning the second question the answer is negative in general. On the first question, a state $\omega$ is called multiplicative if $\omega(xy)=\omega(x)\omega(y)$ for all $x,y\in A$. Note that the notion of positivity defines a partial order on the space of functionals making it an ordered vector space [KR97]. The set of states, $\mathscr{S}$, is convex in the usual sense that for numbers $p_{i}\geq 0,\;\sum_{i=1}^{k}p_{i}=1$ and states $\omega_{i},\;i=1,\dotsc,k$ the functional $\sum_{i}p_{i}\omega_{i}$ is also a state. The extreme points of $\mathscr{S}$ are called pure states. In the case of abelian $C^{*}$ algebras a state is pure if and only of it is multiplicative [KR97]. Thus in a pure state any two observables are uncorrelated. This is not generally true in the non-abelian quantum case.

Next we come to the important notion of independence. First, given $S\subset A$ let $A(S)$ denote the subalgebra generated by $S$ (the smallest subalgebra of $A$ containing $S$). Two subsets $S_{1},S_{2}\subset A$ are defined to be independent if all the pairs $\{(x_{1},x_{2}):x_{1}\in A(S_{1}),x_{2}\in A(S_{2})\}$ are uncorrelated. As independence and correlation depend on the state we sometimes write $\omega$-independent/uncorrelated when to emphasize this. Clearly, independence is much stronger condition than being uncorrelated. It is easy to construct examples in 3 or more dimensions where a pair of observables $x,y$ are uncorrelated but they are not independent: for example, $x^{2}\text{ and }y$ maybe correlated. However, in 2 dimensions $x\text{ and }$ are uncorrelated if and only if one of them is 0 or $c\mathbbm{1}$. Let us note that as in the quantum case two dimensions is an exceptional case. The next theorem shows the structural implications of independence.

Theorem 3**.**

Two sets of observables $S_{1},S_{2}$ in a finite dimensional abelian $C^{*}$ algebra $A$ are independent in a state $\omega$ if and only if for the (unital) subalgebras $A(S_{1})$ and $A(S_{2})$ generated by $S_{1}$ and $S_{2}$ respectively there exist states $\omega_{1}\in{\mathscr{S}}(A(S_{1})),\;\omega_{2}\in{\mathscr{S}}(A(S_{2}))$ such that $(A(S_{1})\otimes A(S_{2}),\{\omega_{1}\otimes\omega_{2}\})$ is a cover of $(A(S_{1}S_{2}),\omega^{\prime})$ where $A(S_{1}S_{2})$ is the subalgebra generated by $\{S_{1},S_{2}\}$ and $\omega^{\prime}$ is the restriction of $\omega$ to $A(S_{1}S_{2})$.

Proof.

First assume that $S_{1}=\{x\}$ and $S_{2}=\{y\}$. Let $\{x_{1},\dotsc,x_{n}\}$ be an atomic basis of $A$. Let $x=\sum_{i}a_{i}x_{i}$ and $y=\sum_{i}b_{i}x_{i}$. Some of these coefficients may be 0 and some may be equal. Write

[TABLE]

Here the $a_{i}$’s are distinct the $P_{i}=x_{i_{1}}+x_{i_{2}}+\cdots+x_{i_{r}}$ corresponding to all basis elements whose coefficients are equal to $a_{i}$. Similarly for $Q_{j}$’s. Note that $P_{i}P_{m}=\delta_{im}$ and $Q_{j}Q_{s}=\delta_{js}$. By Lagrange interpolation there are polynomials $f_{i}(\lambda),\;i=1,\dotsc,k$ and $g_{j},\;j=1,\dotsc,l$ such that $f_{i}(a_{r})=\delta_{ir}$ and $g_{j}(b_{s})=\delta_{js}$. Since $x,y$ are $\omega$-independent

[TABLE]

The subalgebra $A(S_{1})$($A(S_{2})$) is generated by the $P_{i}$’s($Q_{j}$’s). Clearly $\{P_{i}:i=1,\dotsc,k\}$ and $\{Q_{j}:j=1,\dotsc,l\}$ are atomic bases for $A(S_{1})$ and $A(S_{2})$ respectively. Define states $\omega_{1}$ and $\omega_{2}$ of $A(S_{1})$ and $A(S_{2})$ resp. by restricting $\omega$ to these subalgebras. Let $\phi:X\otimes Y\rightarrow A^{\prime}$ be the natural map $\phi(u\otimes v)=uv$. Using equation 4 it is a routine check that $(A(S_{1})\otimes A(S_{2}),\{\omega_{1}\otimes\omega_{2}\})$ is a cover of $(A(S_{1},S_{2}),\omega^{\prime})$.

Now for the general case. Since $A(S_{1})$ and $A(S_{2})$ are subalgebras of $A$ they have atomic bases $\{u_{i}\}$ and $\{v_{j}\}$ respectively. As in the previous case we have polynomials $\{p_{i}\}\text{ and }\{q_{j}\}$ in several variables such that $p_{i}(x_{1},\dotsc,x_{k_{i}})=u_{i}$ and $q_{j}(y_{1},\dotsc,y_{m_{j}})=v_{j}$ where $x_{i}\in S_{1}\text{ and }q_{i}\in S_{2}$. We do not have easy interpolating polynomial in this case. By repeating the argument of the singleton case above we get the appropriate cover and complete the proof.

The converse is clear from the definition of a cover and the fact that in a product state $\omega_{1}\otimes\omega_{2}(z_{1}\otimes z_{2})=\omega_{1}(z_{1})\omega_{2}(z_{2})$. ∎

We can even extend it to infinite tensor product by restricting to finite segments. The next step is to extend the notion of independence to more than two subsets. Let $S_{1},\dotsc,S_{k}\subset A$ and $\omega$ a state of $A$. Then the subsets are defined to be $\omega$-independent if for all $x_{i}\in A(S_{i}),\;i=1,\dotsc,k$ we have

[TABLE]

Here $A(S_{i})$ is the subalgebra generated by $S_{i}$. We can then show that for states $\omega_{i}\in{\mathscr{(}}A(S_{i}))$, the restriction of $\omega$ to $A(S_{i})$ the pair $(A(S_{1})\otimes\cdots\otimes A(S_{k}),\omega_{1}\otimes\cdots\otimes\omega_{k})$ is a cover of $A(S_{1}\dotsc S_{k}),\omega^{\prime}$, where $\omega^{\prime}$ is the restriction of $\omega$ to $A(S_{1}\dotsc S_{k})$, the algebra generated by $S_{1},\dotsc,S_{k}$. We thus see the relation between independence and (tensor) product states in the classical or commutative theory. The non-commutative or quantum case is more delicate and requires careful handling.

3.2 Probability distribution functions

In this section we investigate another important concept of a (cumulative) distribution function (d.f) in the algebraic framework. As the paper’s primary concern is an alternative formulation of mathematical models of information and communication we do not undertake an extensive exploration of the algebraic approach to probability concepts. However, the notion of a distribution function underpins large part of probability theory and its applications. One of the advantages of using $C^{*}$ or more general Banach algebra is that we have both algebraic and analytical methods at our disposal.

Given a subalgebra $B\subset A$ of an abelian $C^{*}$ algebra let $S_{a}=\{x\in A:xs=0\;\forall s\in S\}$ be the annihilator of $S$. This is an ideal666An ideal of a algebra $A$ is a subset $I$ of $A$ which is closed under addition and for every $x\in A$, $xI\subset I$. Hence a non-zero proper ideal cannot contain the identity of $A$ and hence there is an approximate identity. An approximate identity in an ideal $B$ is a net $\{y_{\lambda}\}$ with $0\leq y_{\lambda}\leq\mathbbm{1}$ such that $xy_{\lambda}\rightarrow x$ (also $yx_{\lambda}\rightarrow x,\;\forall x\in B$ if the algebra is nonabelian). For the details see [KR97]. Obviously $S_{a}$ cannot contain the identity of the original algebra unless $S=\{0\}$. We only mention that nets [Kel75] are generalization of sequences where the indexing set is not required to be countable. However, in the case of separable algebras (algebras with a dense countable set) the reader may substitute “sequence” for “net”. In the following it will suffice for our purpose to restrict to the separable case although we often use the language of “nets”. We can now define distribution of a set of observables.

Definition 2**.**

Let $S=\{x_{1},x_{2},\dotsc,x_{n}\}$ be a finite self-adjoint subset of $A$ where $(A,\omega)$ is a fixed probability algebra. For ${\tt t}=(t_{1},t_{2},\dotsc,t_{n})\in\mathbb{R}$ let $S_{\tt t}\subset A$ denote the set of elements $\{(t_{i}\mathbbm{1}-x_{i}):i=1,\dots,n\}$ and $S_{\tt t}^{-}$ the set of elements $\{z_{-}:z\in S_{\tt t}\}$, negative parts of members of $S_{\tt t}$. Let $\{e_{\lambda}({\tt t})\}$ be approximations of identity in the annihilator ideal $(S^{-}_{\tt t})_{a}$. Then the $\omega$-distribution of $S$ is defined to be the real function

[TABLE]

The rationale for this definition is simple. For convenience, restrict to a single random variable. Suppose $X$ is a bounded random variable on a probability space $\{\Omega,{\mathcal{S}},P\}$. Then the distribution function $f(t)=P(\{\alpha\in\Omega:X_{t}=tI-X(\alpha)\geq 0\})$. For a fixed $t$ write the random variable $X_{t}=X_{t+}-X_{t-}$ as the difference of two non-negative random variables. Then the distribution function of $X$ is the probability of the event $E_{t}$ where $E_{t}=\{\alpha\in\Omega:X_{t}(\alpha)\geq 0\}$. Consider now $X_{t-}$ and $G_{t}=\{\alpha:X_{t}(\alpha)<0\}=\Omega-E_{t}$. Then $X_{t-}$ is $>0$ on $G_{t}$ and 0 outside it. If $Y$ is any function on $\Omega$ such that $YX_{t-}=0$ then $Y$ must vanish on $G_{t}$. Conversely any function $Y$ that vanishes on $G_{t}$ satisfies the equation $YX_{t-}=0$. In particular the indicator function ${\mathscr{I}}_{F_{t}}$ satisfies it. The function ${\mathscr{I}}_{F_{t}}$ is the identity on $(X_{t-})_{a}$ and its expectation value $\int{\mathscr{I}}_{F_{t}}\mathrm{d}P=P(F_{t})$. Although, the indicator functions are not generally continuous we can approximate them by a sequence of continuous functions. This sequence is an approximate identity in the $C^{*}$ algebra of continuous functions. In most cases of interest to us the algebras will be separable. Then the nets can be replaced by sequences. Note that since the net $\{e_{\lambda}\}$ is bounded and increasing the net $\{\omega(e_{\lambda})\}$ converge. Finally, let us observe that even though the approximate identity is not unique the distribution function as defined above is unique. To prove this $\{e_{\lambda}\},\{f_{\lambda}\}$ are two approximate identities. Then using the fact $\omega(e_{\lambda}f_{\mu}-e_{\lambda^{\prime}}f{\mu^{\prime}})=\omega(f_{\mu}(e_{\lambda}-e_{\lambda^{\prime}})+e_{\lambda^{\prime}}(f_{\mu}-f_{\mu^{\prime}}))$ is Cauchy since $f_{\mu}(e_{\lambda}-e_{\lambda^{\prime}})\rightarrow(e_{\lambda}-e_{\lambda^{\prime}})$ and $e_{\lambda^{\prime}}(f_{\mu}-f_{\mu^{\prime}})\rightarrow f_{\mu}-f_{\mu^{\prime}}$ we conclude that the double-net $\{\omega(e_{\lambda}f_{\mu})\}$ converges to the limit $\lim_{\lambda}\omega(e_{\lambda})=\lim_{\mu}\omega(f_{\mu})$. Extending the definition of the d.f to an arbitrary element $z$ in the algebra is simple. Write $z=x+iy$ where $x$ and $y$ are self-adjoint. Let $F_{x}(t)\text{ and }F_{y}(t)$ denote the d.f of $x$ and $y$ respectively. Then the d.f of $z$: $F_{z}(t)=F_{x}(t)+iF_{y}(t)$.

Theorem 4**.**

Let $x_{1},\dotsc,x_{n}$ be self-adjoint elements of an abelian $C^{*}$ algebra $A$. Let $F(t_{1},\dotsc,t_{n})$ be their joint distribution function. Then $F(t_{1},\dotsc,t_{n})$ is non-negative, left-continuous and non-decreasing in each variable. We also have boundary conditions

[TABLE]

If the elements are independent and $F(t_{i})$ denotes the distribution function of $x_{i}$ then

[TABLE]

If a sequence $x_{n}\rightarrow x$ in then the corresponding d.f’s $F_{x_{n}}(t)\rightarrow F_{x}(t)$.

Proof.

This is of course a standard result in probability theory. We sketch an algebraic proof in the current setting. The most direct approach is to use the notion of continuous function calculus which essentially asserts that continuous functions on the spectrum can be lifted to define functions on the algebra. More precisely, given an element $x\in A$ there is an isometric algebra homomorphism between the algebra of continuous functions on the spectrum of $x$, $C(\text{sp}(x))$ and the closed subalgebra $C(x)$ generated by $x$ [KR97]. Thus for every function $f(u)$ on $\text{sp}(x)$ there is a unique element $f(x)$ in $C(x)$ such that if $f(u)\geq 0$ then $f(x)\geq 0$. Since for any real $c$ and $\delta>0$ , $|t+\delta-u|-(t+\delta-u)\leq|t-u|-(t-u)$ we infer that $|t+\delta-x|-(t+\delta-x)\leq|t-x|-(t-x)$ for self-adjoint $x\in A$. Now for any $y\in A$ if $xy=0$ then $|x|y=0$ and hence $x_{+}y=x_{-}y=0$. So if $x\leq z$ and $v\in A$ then $zv=0$ implies $xv=0$. Thus the annihilator ideal of $|t+\delta-x|-(t+\delta-x)$ contains the annihilator ideal of $|t-x|-(t-x)$. The continuity follows from the following construction which is useful for calculating distributions. Write $x(t)=t\mathbbm{1}-x$, ${\tt t}=(t_{1},t_{2},\dotsc,t_{n})$ and $\chi({\tt t})=x_{1}(t_{1})_{+}\times x_{2}(t_{2})_{+}\times\dotsb\times x_{n}(t_{n})_{+}$. For integer $m>0$ let

[TABLE]

where ${\tt t}+1/m=(t_{1}+1/m,t_{2}+1/m,\dotsc,t_{n}+1/m)$. Although $e_{m}({\tt t}+1/m)$ is not a member of the annihilating ideal $S^{-}({\tt t})_{a}$ of $S^{-}({\tt t})$ it belongs to $S^{-}({\tt t}+1/m)_{a}\supset S^{-}({\tt t})_{a}$. Let $e_{\lambda}({\tt t})$ be an approximate identity in $S^{-}({\tt t})_{a}$. One can show using the Gelfand representation that

[TABLE]

We omit the details but the reader can convince herself by taking an algebra of functions.

This implies the first part of the theorem. To prove the boundary conditions we use the fact that the spectrum of any element $x\in A$ is bounded by $|\!|x|\!|$. Hence, for $t<-|\!|x|\!|$, $t\mathbbm{1}-x$ has a strictly negative spectrum. Then $(t\mathbbm{1}-x)_{-}=-(t\mathbbm{1}-x)$ is invertible and its annihilator ideal consists of [math] alone. Consequently, $F(t,\dotsc,)=0$ for all $t<-|\!|x|\!|$. The other extreme case is proved similarly, $t\mathbbm{1}-x$ being strictly positive for $t>|\!|x|\!|$. Finally, suppose the elements $\{x_{1},x_{2}\dotsc,x_{n}\}$ are independent. Since $x_{+}$ lies in the closed subalgebra generated by $x$ the definition of independence and equation 5 implies that the joint distribution function is a product. One proves the last statement using a sequence like (5). ∎

We see that, starting from a purely algebraic definition of independence and distributions we can recover their essential properties. In particular, for algebras which are finite or infinite tensor product of finite-dimensional algebras we have the following.

Proposition 2**.**

Let $A$ be a finite-dimensional abelian $C^{*}$ algebra. Let $x\in\otimes^{\infty}A$ and $x_{a}$ its annihilating ideal. Suppose $x$ is a finite sum. Then there is a unique (up to permutation) decomposition

[TABLE]

Further, there exist polynomials without constant term $g_{i}$ such that $P_{i}=g_{i}(x)$. Thus, $x=\sum_{i}a_{i}g_{i}(x)$. Then $x_{a}$ has an identity $\mathbbm{1}-\sum_{i}P_{i}$.

Proof.

Since $x$ is finite sum it may be considered as an element of $\otimes^{n}A$ for some finite $n$. The space $\otimes^{n}A$ has a finite atomic basis, say, $\{Y_{1},\dotsc,Y_{m}\}\;(m=2^{\text{dim}(A)})$. Let $x=\sum_{i=1}^{m}a_{i}Y_{i}$ and let $J=\{i:a_{i}=0\}$. Then $x=\sum_{i\notin J}a_{i}Y_{i}$. Let $P_{i}$ be the sum of all $Y_{i}$ for which the coefficients $a_{i}$ are equal. then $x=\sum_{i}a_{i}P_{i}$ with $a_{i}$ distinct and non-zero. Next use Lagrange interpolation to obtain polynomials $g_{i}$ such that $g_{i}(0)=0$ and $g_{i}(a_{j})=\delta_{ij}$. To prove uniqueness let $x=\sum_{j}b_{j}Q_{j}$ be another such decomposition. Then $xP_{i}Q_{j}=a_{i}P_{i}Q_{j}=b_{j}P_{i}Q_{j}$. Since $\sum_{i}P_{i}x=x$ for a fixed $i$ there must be at least one $j_{i}$ with $P_{i}Q_{j_{i}}\neq 0$ then $a_{i}=b_{j_{i}}$. There cannot be more than one such $j_{i}$ since the $b_{j}$’s are distinct. Arguing in the reverse direction we conclude that $i\leftrightarrow j_{i}$ is a permutation. The last statement follows trivially. ∎

Let $x=\sum_{i}a_{i}P_{i}$ be as in the proposition. We call this the spectral decomposition of $x$. If $\omega$ is a state define

[TABLE]

The map ${\mathscr{I}}_{\omega}(x)$ can be considered as a “centroid” of the possible outcomes of measurement of $x$. We can extend the proposition to arbitrary element in ${\mathscr{A}}=\otimes^{\infty}A$ by using a sequence of finite-dimensional projections as above to approximate. However, the proposition suffices for most of our requirements. Now let

[TABLE]

$Z$ may not be a member of ${\mathscr{A}}$ in general as we treat the above as a formal sum. However, we suppose that for real $t$, $(t\mathbbm{1}-Z)_{+}=(|t\mathbbm{1}-Z|+(t\mathbbm{1}-Z))/2$ can be expressed as finite sum. We will see an example below. Then the required identity is given as follows. It is clear that for $\delta>0$ small enough $|(t+\delta)\mathbbm{1}-Z|+((t+\delta)\mathbbm{1}-Z)=\sum_{k}a_{k}Y_{k}:a_{k}>0$ is finite sum where $Y_{k}$ constitute an atomic basis. Let $P_{\delta}=\sum_{k}Y_{k}$. Then the required identity is given by $P_{0}=\lim_{\delta\rightarrow 0}P_{\delta}$. This is essentially a variant of equation 5 in Theorem 4.

3.3 Examples

In this section we consider some examples from standard probability theory. It will be demonstrated that the algebraic approach not only gives a different perspective on some familiar situations it can also provide additional computational tools. First, we review the correspondence between some concepts from the standard theory with our algebraic model. An event in probability theory is a measurable subset of the probability space. The random variable characterizing any (measurable) subset $S$ is its indicator function $I_{S}$. In the algebraic language it is a projection $Q_{S}$. The probability of the event corresponds to the expectation value $\omega(Q_{S})$ of the projection. In the cases we consider the projections will generally exist in the algebra itself. In some cases we consider infinite formal sums which are not in the algebra but any finite segment of the sum do belong to the algebra. In the actual computation we always use a “cut-off” to restrict to such a finite segment. In the cases where projections are not members of the algebra we can find a sequence (or net) that “converges in the mean” to the appropriate projection or indicator function. This situation generally arises in the continuous case which is only touched upon peripherally.

Binomial distribution. Consider again infinite sequences of Bernoulli trials as in the second example of the previous section. We can think of coin-tossing with “heads” signaling success. Let $Z$ be the observable (random variable) corresponding to the number of success. What is its d.f.? Let $n,k$ be a positive integers with $k<n$. We want to find the distribution $F(k:n)$ of $Z$. Recall that $G$ is the 2-dimensional algebra and let $A=\otimes^{n}G$. Let $\{y_{0},y_{1}\}$ be the atomic basis of $G$ with $y_{1}$ corresponding to success. Set

[TABLE]

Here ${\mathcal{S}}$ denotes the distinct permutations of the factors in the tensor product. Thus, the $r$th term $Y_{r}$ is the sum of all $\binom{n}{r}$ products with $r$ $y_{1}$’s. Its value is $r$. Note that $Y_{r}Y_{s}=\delta_{rs}$. We have

[TABLE]

In this case the identity in the annihilator ideal of $U$ exists and is given by the projection operator $P=\sum_{r=0}^{k}Y_{r}$. Since the Bernoulli $F(k:n)=\Omega(P)=\sum_{0}^{k}\binom{n}{k}p^{k}(1-p)^{n-k}$. Note that we can easily find the distribution in states where the observables are not independent. 2. 2.

Waiting time. Let us start with a simple version of the problem of waiting time. Suppose we have a binary source with fixed probability distribution emitting a bit per unit time. The waiting time is the time elapsed before the first appearance of 1. It is a random variable or observable $W$ in our formalism. Using the notation above

[TABLE]

This is an unbounded infinite sum and does not belong to the algebra. However, for any $t\geq 0$,

[TABLE]

is finite (of course, $F_{W}(t)=0\text{ for }t<0$). Here $\lfloor t\rfloor$ is the largest integer $\leq t$. Using the trick explained before the examples we replace $t$ by $t+\delta$ (this is to take into account the case when $t$ is an integer). The required projection (approximate identity) is

[TABLE]

The distribution function in a state $\Omega$ is given by $F_{W}(t)=\Omega(P_{W}(t))$. If $\Omega=\omega\otimes\omega\otimes\cdots$ is an infinite product state with $\omega(y_{1})=p=1-\omega(y_{0})$ then $F_{W}(t)=\sum_{k=0}^{\lfloor t\rfloor}p(1-p)^{k}$.

Next we generalize the problem of waiting time to arbitrary strings. Explicitly, given a string $\xi$ of length $n$ the waiting time is the time before a contiguous stream of bits matching $\xi$ appears. The preceding case is for $\xi=1$. We will only construct the observable corresponding to waiting time $W$ in this general case. It gives a nice illustration of the algebraic techniques. Let $X$ be the tensor representation of $\xi$. Waiting time 0 corresponds to the observable $X\otimes\mathbbm{1}$. We use the following notation. Write $\mathbbm{1}_{1}$ for the identity in the 2-dimensional space $G$ and $\mathbbm{1}_{k}=\mathbbm{1}_{1}\otimes\mathbbm{1}_{1}\otimes\cdots\mathbbm{1}_{1}$, the $k$-fold tensor product. The symbol $\mathbbm{1}$ (without subscripts) will be reserved for the identity in $\otimes^{\infty}A$. The element $Y_{0}=X\otimes\mathbbm{1}$ corresponds to waiting time 0: the first $n$ symbols received match the given string. We expect the element corresponding to waiting time 1 will be “proportional” to $Y_{1}^{\prime}=\mathbbm{1}_{1}\otimes X\otimes\mathbbm{1}$. Although $Y_{0}$ and $Y_{1}^{\prime}$ are projections they need not be orthogonal in the sense $Y_{1}^{\prime}Y_{0}=0$. So they do not correspond to mutually exclusive events. Recall that when interpreted as functions on some measure space projections are indicators of measurable sets (events). We therefore adopt an orthogonalization scheme similar to Gram-Schmidt. The observable $Y_{1}=Y_{1}^{\prime}-Y_{1}^{\prime}Y_{0}$ is projection and satisfies $Y_{1}Y_{0}=0$. Viewed as a function it takes value 1 only when the input string is of the $\zeta=b_{0}\xi\ldots$ and such that the prefix of length $n$ of $\zeta$ does not match $\xi$. It corresponds to waiting time 1. Defining inductively, let

[TABLE]

It is easily verified that $Y_{j}Y_{k}=\delta_{jk}Y_{k}$. The element $W=\sum_{k=0}^{\infty}kY_{k}$ corresponds to the waiting time in this case. Again it is not an element of the algebra but $|t\mathbbm{1}-W|+t\mathbbm{1}-W$ is. 3. 3.

Markov Chains. We define a discreet time Markov chain on an observable algebra $(A,\omega)$ as a sequence of positive and unital maps $\{\phi_{0},\phi_{1},\dotsc,\}$ and an initial element $x_{0}\in A$. Let us confine to discrete chains. Let ${\mathcal{A}}=\{x_{1},x_{2},\dotsc,\}$ be a fixed atomic basis. A chain-state is a sequence $\{z_{0},z_{1},\dotsc,\}$ where each $z_{i}\in{\mathcal{A}}$. The usual term for what we call chain-state is simply “state” but the latter has a very specific meaning in operator algebras. Let $\xi_{n}=\{z_{0},z_{1},\cdots,z_{n}\}$ be a finite segment of the chain-state. We are interested in the transition from $x_{0}$ to $x_{n}$ via the path $\xi_{n}$. The transition probability is defined recursively as follows.

[TABLE]

Let us examine this definition in the special case of stationary Markov chains. A Markov chain is defined to be stationary if all the transition maps are identical: $\phi_{0}=\phi_{1}=\phi_{2}=\dotsb$. For a stationary chain

[TABLE]

Here $\phi(i,j)$ is the $(ij)$th matrix element of $\phi$ with respect to the basis ${\mathcal{A}}$ and $z_{k}=x_{i_{k}}$. This looks very similar to quantum transition probability. In the later case the$x_{i}$ are projections on a Hilbert space. Further, when we consider transitions over all possible paths then we get an analogue of Feynman’s “sum over paths” for total transition probability.

3.4 Limit theorems

The limit theorems of probability theory are important for its theoretical structure as well as its empirical justification. We will be primarily concerned with the bounded case where the proofs are simpler. We state two of these but prove only the weak law of large numbers. From information theory perspective it is perhaps the most useful limit theorem. Let $X_{1},X_{2},\cdots,X_{n}$ be independent, identically distributed (i.i.d) random variables on a probability space $\Omega$ with probability measure $P$. Let $\mu$ be the mean of $X_{1}$ (hence any $X_{i}$). We assume that the the variance $E(X_{1}-\mu)^{2}$ is bounded. Here, $E(X)$ denotes the expectation value of random variable $X$.

• •

Weak law of large numbers. Given $\epsilon>0$

[TABLE]

• •

Central limit theorem. If $0<E(X_{1}^{2})=\sigma<\infty$ then for any real $x$ as $n\rightarrow\infty$

[TABLE]

A few comments about these famous limit theorems. These are statements about different types of convergence [Bil95]. The theorems can be strengthened but since we are dealing with bounded random variables the above formulations suffice. These theorems require assigning of probabilities. All we have at our disposal is the algebra and one or more positive functionals (states) which give us expectation values. But we have already seen how to define probability distribution functions. What we need are appropriate projections or approximations to them. Given a self-adjoint observable $x$ and a real number $a$ write $x-a$ for the element $x-a\mathbbm{1}$. Let $A(x-a)_{+}$ be the (two-sided) ideal generated by the positive part of $x-a$. Let $e_{n}=(x-a)_{+}[(x-a)_{+}+\delta_{n}]^{-1}$ where $0<\delta_{n}$ such that $\lim_{n\rightarrow\infty}\delta_{n}=0$. Then it can be shown that for any $y\in A(x-a)_{+}$, $\lim_{n\rightarrow\infty}e_{n}y\rightarrow y$ in the norm. Hence, $\{e_{n}\}$ is an increasing sequence approximating identity (see [Tak02]). We write $\mathbb{P}(x>a)$ for this approximate identity in $A(x-a)_{+}$. It is not unique but that does not matter since all the limits that we use it to define are independent of the particular choice. The probability corresponding to the “event” $x>a$ is defined to be $P(x>a)=\omega(\mathbb{P}(x>a))=\lim_{n\rightarrow\infty}\omega(e_{n})$. Similarly we can define $P(x<a)=\omega(\mathbb{P}(x<a))$ where $\mathbb{P}(x<a)=\{f_{n}\}$ is an approximate identity in the ideal $A(x-a)_{-}$ obtained by replacing $(x-a)_{+}$ by $(x-a)_{-}$ in $e_{n}$. We can define more complicated events by algebraic operations but it is not necessary for what follows. We also note that although we use probabilistic language in the statements of the results below all the expressions are actually defined in a strictly algebraic setting without reference to any underlying probability space.

Lemma 1** (Chebysev inequality).**

Let $x,y\in A$ be self-adjoint where $(A,\omega)$ is an observable algebra and $y\geq 0$. For any number $\epsilon>0$ we have

[TABLE]

[TABLE]

Proof.

Let $\{e_{n}\}$ be an approximate identity in the ideal $A(y-\epsilon)_{+}$. By definition $e_{n}\leq\mathbbm{1}$. Hence, $\omega(y)=\omega(ye_{n})+\omega(y(\mathbbm{1}-e_{n}))\geq\omega(ye_{n})$. Since $(y-\epsilon)_{-}$ annihilates the ideal $A(y-\epsilon)_{+}$, $\omega(ye_{n})=\omega([y-\epsilon]e_{n})+\omega(\epsilon e_{n})=\omega([y-\epsilon]_{+}e_{n})+\epsilon\omega(e_{n})\geq\epsilon\omega(e_{n})$. Hence, $\omega(y)\geq\epsilon\omega(e_{n})$. Taking limits we obtain the first inequality. Observe that for any $x\in A$, $P(|x|>\epsilon)=P(|x|^{2}>\epsilon^{2})$ for the ideals $A(|x|-\epsilon)_{+}$ and $A(|x|^{2}-\epsilon^{2})_{+}$ coincide. This follows from the identities $|x|^{2}-\epsilon^{2}=(|x|+\epsilon)(|x|-\epsilon)$ and hence $(|x|^{2}-\epsilon^{2})_{+}=(|x|+\epsilon)(|x|-\epsilon)_{+}$ plus the fact that $|x|+\epsilon$ is invertible. Hence the second inequality follows from the first by putting $y=(x-\omega(x))^{2}$ and using $\epsilon^{2}$ in place of $\epsilon$. ∎

We will prove next a convergence result which implies the weak law of large numbers.

Theorem 5** (Law of large numbers (weak)).**

*If $x_{1},\dotsc,x_{n},\dotsc$ are $\omega$-

independent self-adjoint elements in an observable algebra and $\omega(x_{i}^{k})=\omega(x_{j}^{k})$ for all positive integers $i,j\text{ and }k$ (they are identically distributed) then*

[TABLE]

Proof.

We may assume $\mu=0$ (by reasoning with $x_{i}-\omega(x_{i})$ instead of $x_{i}$). First we prove the statement for $k=2$. Then $\omega(|\frac{x_{1}+\dotsb+x_{n}}{n}|)^{2}=\sum_{i}\omega(x_{i}^{2})/n^{2}=\omega(x_{1}^{2})/n$. The first equality follows from independence ($\omega(x_{i}x_{j})=\omega(x_{i})\omega(x_{j})=0\text{ for }i\neq j$) the second from the fact that they are identically distributed. The case $k=2$ is now trivial. Now let $k=2m$. Then $|x_{1}+\dotsb+x_{n}|^{k}=(x_{1}+\dotsb+x_{n})^{k}$. Put $s_{n}=(x_{1}+\dotsb+x_{n})/n$. Expanding $s_{n}^{k}$ in a multinomial series we note that independence and the fact that $\omega(x_{i})=0$ implies that all terms in which at least one of the $x_{i}$ has power 1 do not contribute to $\omega(s_{n}^{k})$. The total number of the remaining terms is $O(n^{m})$. Since the denominator is $n^{2m}$ we see that $\omega(s_{n}^{k})\rightarrow 0$. Since for any $x\in A$, $|x|=(x^{2})^{1/2}$ can be approximated by polynomials in $x^{2}$ we conclude that $\omega(|s_{n}|)\rightarrow 0$. Finally, using the Cauchy-Schwartz type inequality $\omega(|s_{n}|^{2r+1})\leq\omega(s_{n}^{2})\omega(s_{n}^{2r})$ we see that the theorem is true for all $k$. ∎

Corollary 5.1**.**

Let $x_{1},\dotsc,x_{n}\text{ and }\mu$ be as in the Theorem and set $s_{n}=(x_{1}+\dotsb+x_{n})/n$. Then for any $\epsilon>0$ there exist $n_{0}$ such that for all $n>n_{0}$

[TABLE]

Proof.

Using Chebysev inequality we have $P(|s_{n}-\mu|>\epsilon)=P(|s_{n}-\omega(s_{n})|>\epsilon)\leq\frac{\omega(|s_{n}-\mu|^{2})}{\epsilon^{2}}$. As $\omega(|s_{n}-\mu|^{2})\rightarrow 0$ (Theorem 5) there is $n_{0}$ such that $\omega(|s_{n}-\mu|^{2})<\epsilon^{3}$ for $n>n_{0}$. ∎

4 Communication and Information

We now come to our original theme: an algebraic framework for communication and information processes. We can view information as a measure of our state of ignorance or uncertainty. Mathematically, it is equivalent to some measure associated with a probability distribution of some physical quantity which we identify with an observable. Thus any manipulation of the quantity, for example, transmitting it or measuring it is given by some operation on the observable. Since our primary goal is the modeling of information processes we refer to the simple model of communication in the Introduction and model different aspects of it.

4.1 Source and coding

Definition 3**.**

A source is a pair ${\mathscr{S}}=(X,S)$ where $X\subset A$, $A$ a $C^{*}$ algebra and $S$ is a set of states. A source is static if $S$ consists of single state. It is discrete if $X$ is countable.

This definition abstracts the essential properties of a source. A real source could be an animate (human speech, for example) or inanimate object (a radio transmitter, for example). Its output can be considered discrete, for example, a keyboard with a fixed alphabet or continuous like radiation from a star. In this work we will be mainly concerned with discrete sources. Then $X$ will be called the source alphabet. We assume that at each instant there is a probability distribution on the letters of the alphabet characterizing the state of the source at that instant. Thus a discrete source is a countable set of random variables. In the algebraic view it is a sequence of elements $X$ of a $C^{*}$ algebra. The set of states $S$, called the states of the source, provide the probability distributions. If this distribution does not change (equivalently $S$ consists of a single element) then we have a static source. We will mostly deal with static sources in this work. When we model transmission of information as a Markov process the state of the source is identified with the initial probability distribution. There is dual view. Suppose that a source $\mathscr{S}$ emits letters from a finite alphabet. Then the set $X$ in the above definition is a subset of the atomic basis (corresponding to the alphabet) of the algebra $A$. For a state $\omega$ define

[TABLE]

We say that ${\mathcal{O}}_{\omega}$ is the output of the source in state $\omega$. Intuitively, ${\mathcal{O}}_{\omega}$ is a kind of mean “point” in the space of outputs (compare it with the notion of center of mass in mechanics). More importantly, it facilitates calculation of important quantities and has close analogy with the quantum case. The quantum analogue may be pictured as follows. The source outputs “particles” in definite “states” $x_{i}$ with probability $p_{i}=\omega(x_{i})$. Note that here state corresponds to a projection operator. A measurement for $x_{i}$ means applying the dual operator $\omega_{i}\;(\omega_{i}(x_{j})=\delta_{ij})$ giving $\omega_{i}({\mathcal{O}}_{\omega})=p_{i}$.

Let $\mathscr{Z}=(X,\omega)$ be a static discrete source. Suppose every $x\in X$ belongs to a finite-dimensional subalgebra generated by a (finite) set of $\omega$-independent elements. Then using the Theorem 3 we may assume that $A=\bigotimes^{\infty}{B}$ where $B$ is finite-dimensional abelian $C^{*}$ algebra and $\omega$ is an (infinite) product state. In this case, each element of $X$ is a tensor product of elements of an atomic basis of $B$. In the rest of the paper we assume that $X$ is the product basis of atomic elements. For example, if $B$ is the two dimensional algebra with atomic basis $\{y_{0},y_{1}\}$ then $X$ is the set of elements of the form $z_{1}\otimes z_{2}\otimes\dotsb\otimes z_{k}\otimes\mathbbm{1}\otimes\mathbbm{1}\otimes\dotsb$ where $z_{i}\in\{y_{0},y_{1}\}$.

4.2 Source coding

Let $B$ be a finite-dimensional $C^{*}$ algebra and $A=\bigotimes^{\infty}{B}$ . We consider $\otimes^{n}B$ as a subalgebra of $A$ via the standard embedding (all “factors” beyond the $n$th place equal $\mathbbm{1}$). Let $X_{n}$ be its atomic basis in some fixed ordering and let $X=\bigcup_{n}X_{n}$. We can consider $B$ as the source alphabet and $X_{n}$ as strings of length $n$. Let $B^{\prime}$ be another finite-dimensional $C^{*}$ algebra and $A^{\prime}=\bigotimes^{\infty}{B}^{\prime}$. A source coding is a linear map $f:B\rightarrow T=\subset\sum_{k\geq 1}^{m}\otimes^{k}B^{\prime}$. Here $T$ is the linear subspace. It induces a (linear) map

[TABLE]

$\otimes^{n}f$ extends to a unique map $F:A\rightarrow A^{\prime}$. Note that we first induce a map on $\otimes^{n}B$, $n=1,2,\dotsc,$ and then lift it to $A$. We allow the map $f$ to take values that are not simple products. However, for classical communication we require that each atomic basis element $x_{i}\in B$ be mapped to a tensor product of atomic basis elements. Since we are dealing with classical information in this paper it will be implicitly assumed that all the codes are classical. Let us consider an example to clarify these points.

Example. Let $\{x_{0},x_{1},x_{2},x_{3}\}$ be an atomic basis for $B$. Let $B^{\prime}=G$ with atomic basis $\{y_{0},y_{1}\}$. Define $f_{1}$ by $f_{1}(x_{0})=y_{0},f_{1}(x_{1})=y_{1},f_{1}(x_{2})=y_{0}\otimes y_{1}\text{ and }f_{1}(x_{3})=y_{1}\otimes y_{0}$. Denote by $\hat{f}_{1}$ its extension to tensor products. Since $\hat{f}_{1}(x_{0}\otimes x_{1})=y_{0}\otimes y_{1}=\hat{f}_{1}(x_{2})$, $\hat{f}_{1}$ is not injective. Hence it cannot be inverted on its range. Consider next the map $f_{2}(x_{0})=y_{0},f_{2}(x_{1})=y_{0}\otimes y_{1},f_{2}(x_{2})=y_{0}\otimes y_{1}\otimes y_{1}\text{ and }f_{2}(x_{3})=y_{1}\otimes y_{1}\otimes y_{1}$. This map is invertible but one has to look at the complete product before finding the inverse. It is not prefix-free.

Now going back to the general formulation a code $f:B\rightarrow T$ is defined to be prefix-free if for distinct members $x_{1},x_{2}$ in an atomic basis of $B$, $f^{\prime}(x_{1})f^{\prime}(x_{2})=0$ where $f^{\prime}$ is the map $f^{\prime}:B\rightarrow\bigotimes^{\infty}{B}^{\prime}$ induced by $f$. That is, distinct elements of the atomic basis of $B$ are mapped to orthogonal elements. Recall that two elements $x,y$ of an algebra are considered orthogonal if their product $xy=0$. 777The use of the term “orthogonal” may be questionable since there is no scalar product. But let us observe that the projection operators corresponding to two pure states in quantum mechanics have algebraic product 0 if and only if they are orthogonal. Now, in the standard formulation an alphabet is a finite set and a code is a map from $Y\rightarrow Z^{+}$ where $Y,Z$ are alphabets and $Z^{+}$ is the set of non-empty finite strings from $Z$. The definition of prefix-free in this case is clear. In the algebraic language the free monoidal structure defined by concatenation is replaced by the tensor structure. Then the “code-word” $z_{1}\otimes z_{1}\otimes\dotsb\otimes z_{k}\otimes\mathbbm{1}\otimes\mathbbm{1}\otimes\dotsb$ is not orthogonal to another $z^{\prime}_{1}\otimes z^{\prime}_{1}\otimes\dotsb\otimes z^{\prime}_{m}\otimes\mathbbm{1}\otimes\mathbbm{1}\otimes\dotsb$ with $k\leq m$ if and only if $z_{1}=z^{\prime}_{1},\dotsc,z_{k}=z^{\prime}_{k}$. We observe that one has to be careful about correspondence between the two approaches. For example, one might be tempted to identify the identity $\mathbbm{1}$ with the empty string but the $\mathbbm{1}$ is the sum of the members of an atomic basis! The binary operation “+” has a relatively lesser role in the classical formalism but it is crucial in the quantum framework (via superposition principle). Our first result is a useful and well-known inequality proved using algebraic techniques.

Lemma 2** (Kraft inequality).**

Let $B$ be an $n$-dimensional abelian $C^{*}$ algebra. Corresponding to a finite sequence $k_{1}\leq k_{2}\leq\dotsb\leq k_{m}$ of positive integers let $\alpha_{1},\dotsc,\alpha_{m}$ be a set of prefix-free elements in $\sum_{i\geq 1}\otimes^{i}B$ such that $\alpha_{i}\in\otimes^{k_{i}}B$. Further, suppose that each $\alpha_{i}$ is a tensor product of elements from a fixed atomic basis of $B$. Then

[TABLE]

Proof.

Let $\mathbbm{b}=\{y_{1},\dots,y_{n}\}$ be the fixed atomic basis of $B$ and set $k_{m}=M$. We can then restrict our attention to the finite-dimensional algebra $Z=\sum_{i=1}^{M}\otimes^{i}B$. Let $\alpha_{1}=z^{1}_{1}\otimes\dotsb\otimes z^{1}_{k_{1}}\otimes\mathbbm{1}\otimes\dotsb\otimes\mathbbm{1}$ where $z^{1}_{i}\in\mathbbm{b}$. Let $\beta=z^{1}_{1}\otimes\dotsb\otimes z^{1}_{k_{1}}$ and

[TABLE]

Then $Z_{1}\subset\otimes^{M}B$ is a subalgebra (without unit) of dimension $n^{M-k_{1}}$. The assumption that $\alpha_{i}$ are prefix-free implies $\alpha_{2},\alpha_{3},\dotsc,\alpha_{M}$ must be in $Z_{1}^{\prime}$ the “orthogonal” complement to $Z_{1}$ in $Z$. Dimension of $Z^{\prime}_{1}=n^{M}-n^{M-k_{1}}$. Repeating this argument with $\alpha_{2},\dotsc,\alpha_{k_{m-1}}$ we conclude that $\alpha_{k_{m}}$ must be in a subspace of dimension $n^{M}-n^{M-k_{1}}-n^{M-k_{2}}-\dotsb-n^{M-k_{m-1}}$. Since $\alpha_{k_{m}}$ is non-zero $n^{M}-n^{M-k_{1}}-n^{M-k_{2}}-\dotsb-n^{M-k_{m-1}}\geq 1$. This is equivalent to the relation (6). ∎

With the notation of the lemma we call the sequence $W=\{\alpha_{1},\dotsc,\alpha_{m}\}$ decipherable if the tensor product of any two distinct finite ordered sequence of elements from $W$ are distinct. The sequences may have repeated elements. The Kraft inequality is valid for decipherable sequences [Mac53]. However, the proof is essentially combinatorial. The Kraft inequality also provides a sufficiency condition for prefix-free code [Ash90, CT99]. Thus the existence of a decipherable code of word-lengths $(k_{1},k_{2},\dotsc,k_{m})$ implies the existence of a prefix-free code of same word-lengths. In the following, we restrict ourselves to prefix-free codes. If $g:A\rightarrow\otimes^{\infty}B$ is a prefix-free code then it maps orthogonal elements to orthogonal elements. It is therefore an algebra isomorphism (a one-to-one homomorphism). Next we have a technical lemma that is useful in finding bounds.

Lemma 3**.**

Let $f$ be a continuous real function on $(0,\infty)$ such that $xf(x)$ is convex and $\lim_{x\rightarrow 0}xf(x)=0$. Let $A$ be a finite-dimensional $C^{*}$ algebra with atomic basis $\{x_{1},\dotsc,x_{n}\}$ and $\omega$ a state on $A$. Then for any set of numbers $\{a_{i}\;:i=1,\dotsc,n;\;a_{i}>0\text{ and }\sum_{i}a_{i}\leq 1\}$ we have

[TABLE]

Proof.

Let $\omega(x_{i})=p_{i}$. We have to show that $\sum p_{i}f(p_{i}/a_{i})\geq f(1)$. First assume that all $p_{i}>0$ and $\sum_{i}a_{i}=1$. Then

[TABLE]

by convexity of $xf(x)$. The general case can be proved by starting with $a_{i}$ corresponding to $p_{i}>0$ and adding extra $a_{j}$’s to satisfy $\sum_{i}a_{i}=1$ if necessary. The corresponding $p_{j}$ is set to [math]. Now define a new function $g(x)=xf(x),\;x>0$ and $g(0)=0$. The conclusion of the lemma follows by arguing as above with $g$. ∎

Using the lemma for the function $f(x)=\log{x}$ and Lemma 2 we easily deduce the following.

Proposition 3** (Noiseless coding).**

Let $\mathscr{S}$ be a source with output ${\mathcal{O}}_{\omega}\in A$, a finite-dimensional $C^{*}$ algebra with atomic basis $\{x_{1},\dotsc,x_{n}\}$ (the alphabet). Let $g$ be prefix-free code such that $g(x_{i})$ is a tensor product of $k_{i}$ members of the code basis. Then

[TABLE]

Next we give a simple application of Theorem 5. First define a positive functional ${\tt Tr}$ on a finite dimensional abelian $C^{*}$ algebra $A$ with an atomic basis $\{x_{1},\dotsc,x_{d}\}$ by ${\tt Tr}=\omega_{1}+\dotsb+\omega_{d}$ where $\omega_{i}$ are the dual functionals. It is clear that ${\tt Tr}$ is independent of the choice of atomic basis. Informally, the function ${\tt Tr}$ gives the dimension of a projection.

Theorem 6** (Asymptotic Equipartition Property (AEP)).**

Let ${\mathscr{S}}$ be a source with output ${\mathcal{O}}_{\omega}=\sum_{i=1}^{d}\omega(x_{i})x_{i}$ where $\omega$ is a state on the finite dimensional algebra with atomic basis $\{x_{i}\}$. Then given $\epsilon>0$ there is a positive integer $n_{0}$ such that for all $n>n_{0}$

[TABLE]

where $H=\omega(\log_{2}({\mathcal{O}}_{\omega}))$ is the entropy of the source and the probability distribution is calculated with respect to the state $\Omega_{n}=\omega\otimes\dotsm\otimes\omega$ ($n$ factors) of $\otimes^{n}A$. If $Q$ denotes the identity in the subalgebra generated by $(\epsilon I-|\log_{2}(\otimes^{n}{\mathcal{O}}_{\omega})+nH|)_{+}$ then

[TABLE]

Before proving the theorem some explanations are necessary. First $\log_{2}{x}\;(=\ln{x}/\ln{2})$ is usually defined for strictly positive elements of a $C^{*}$ algebra888Henceforth $\log$ will be always with respect to base 2 unless specified otherwise. We extend the definition to all non-zero $x\geq 0$. The standard method of extending complex functions (continuous or analytic) functions to a $C^{*}$ algebra is via functional calculus [KR97]. However, in our case it is simpler. Let $\{y_{i}\}$ be a atomic basis in an abelian $C^{*}$ algebra. Let $y=\sum_{i}a_{i}y_{i}$ with $a_{i}\geq 0$. Then define $\log_{2}{y}=\sum_{i}b_{i}y_{i}$ where $b_{i}=\log{a_{i}}$ if $a_{i}>0$ and 0 otherwise. This definition implies that some standard properties of $\log$ are no longer true (e.g. $2^{\log{x}}\neq x$). But in the present context it gives the correct result when we take expectation values as in the formulas in the theorem. A somewhat longer but mathematically better justified route is to “renormalize” the state. Thus if $\omega(x_{i})=0$ for $k$ indices we define $\omega^{\prime}(x_{i})=\delta$ where $\delta$ is arbitrarily small but positive and $\omega^{\prime}(x_{j})=\omega(x_{j})-k\delta$ where $\omega^{\prime}(x_{j})>k\delta$. If we can prove the theorem now for $\omega^{\prime}$ and since the relations are valid in the limit $\delta\rightarrow 0$ then we are done. We will not take this path but implicitly assume that the probabilities are positive. Finally, note that the element $Q$ is a projection on the subalgebra generated by $(\epsilon I-|\log_{2}(\otimes^{n}{\mathcal{O}}_{\omega})-nH|)_{+}$. It corresponds to the set of strings whose probabilities are between $2^{-nH-\epsilon}$ and $2^{-nH+\epsilon}$. The integer ${\tt Tr}(Q)$ is simply the cardinality of this set.

Proof of the theorem.

First note that $\log{ab}=\log a+\log b$ for elements $a,b\geq 0$ in $A$. We can write $\otimes^{n}{\mathcal{O}}_{\omega}=X_{1}X_{2}\dotsb X_{n}$ where $X_{i}=\mathbbm{1}\otimes\mathbbm{1}\otimes\dotsm\otimes{\mathcal{O}}_{\omega}\otimes\mathbbm{1}\otimes\dotsm\otimes\mathbbm{1}$ with $\log{{\mathcal{O}}_{\omega}}$ in the $i$th place. The fact that $\Omega_{n}$ is a product state on $\otimes^{n}A$ (corresponding to a source whose successive outputs are independent) implies that $X_{i}$ are independent and identically distributed. We can now apply the corollary to Theorem 5 yielding $P(|\log{(\otimes^{n}{\mathcal{O}}_{\omega})}-\Omega_{n}(\log{X_{1}})|>\epsilon)=P(|\log{(\otimes^{n}{\mathcal{O}}_{\omega})}-\omega(\log{({\mathcal{O}}_{\omega})})|>\epsilon)$. ∎

4.3 Communication Channels

Every form of communication requires channels through which signals are sent and received. It is perhaps the most important component in the mathematical models of communication. We will not be dealing with real channels which are complex physical objects— the atmosphere, a telephone cable, a bus on the mainboard of a computer are some examples. Our object is to give simple mathematical models of a channel which still yield interesting results relevant for concrete channels. The original paper of Shannon characterized channels by a transition probability function. Thus, the channel (precisely a two-way channel) has an input alphabet $X$ and output alphabet $Y$ and a sequence of random functions $\phi_{n}:X^{n}\rightarrow Y^{n}$. The latter are characterized by probability distributions $p_{n}(y^{(n)}|x^{(n)})$, the interpretation being: $\phi_{n}(x^{(n)})=y^{(n)}$ with conditional probability $p_{n}(y^{(n)}|x^{(n)})$. Note that the distribution depends on the entire history. We say that such a channel has (infinite) memory. A channel has finite memory if there is an integer $k\geq 0$ such that if $x^{(n)}=x_{n}x_{n-1}\cdots x_{n-k+1}\dotsc x_{1}$ then $p_{n}(y^{(n)}|x^{(n)})=p_{n}(y^{(n)}|x^{\prime(n)})$ for any string $x_{n}^{\prime}$ of length $n$ such that $x_{n}^{\prime}=x_{n},\dotsc,x_{n-k+1}^{\prime}=x_{n-k+1}$. That is, the probability distribution depends on the most recent $k$ symbols seen by the channel. A channel is memoryless if $k=1$. Since we will be dealing mostly with discrete memoryless channels (DMS) this property will be tacitly assumed unless stated otherwise. In the memoryless case it is easy to show the simple form of transition probabilities

[TABLE]

This motivates us to define the channel transformation matrix $C(y_{j}|x_{i})$ with $y_{j}\in Y$ and $x_{i}\in X$. As before in this work $X$ and $Y$ will be finite sets. Since the matrix $C(y_{j}|x_{i})$ is supposed to represent the probability that the channel outputs $y_{j}$ on input $x_{i}$ we must have $\sum_{j}C(y_{j}|x_{i})=1$ for all $i$. In other words, matrix $C(ij)=C(y_{j}|x_{i})$ is row stochastic. This is the standard formulation. [Ash90, CT99, Khi57]999In this work we will not deal with channel coding and decoding. Including these concepts is not difficult but complicates the notation. We now turn to the algebraic formulation. We restrict ourselves to two-terminal channels here.

Definition 4**.**

A DMS channel ${\mathcal{C}}=\{X,Y,C\}$ where $X$ and $Y$ are abelian $C^{*}$ algebras of dimension $m$ and $n$ respectively and $C:Y\rightarrow X$ is a unital positive map. The algebras $X$ and $Y$ will be called the input and output algebras of the channel respectively. Given a state $\omega$ on $X$ we say that $(X,\omega)$ is the input source for the channel.

We recall that a positive map $C:Y\rightarrow X$ is a linear map such that $C(y)\geq 0$ if $y\geq 0$. Sometimes we write the entries of $C$ in the more suggestive form $C_{ij}=C(y_{j}|x_{i})$ where $\{y_{j}\}$ and $\{x_{i}\}$ are atomic bases for $Y$ and $X$ respectively. Thus $C(y_{j})=\sum_{i}C_{ij}x_{i}=\sum_{i}C(y_{j}|x_{i})x_{i}$. Note that in our notation $C$ is an $m\times n$ matrix. Its transpose $C^{T}_{ji}=C(y_{j}|x_{i})$ is the channel matrix in the standard formulation. We have to deal with the transpose because the channel is a map from the output alphabet to the input alphabet. This may be counterintuitive but observe that any map $Y\rightarrow X$ defines a unique dual map ${\mathcal{S}}(X)\rightarrow{\mathcal{S}}(Y)$, on the respective state spaces. Informally, a channel transforms a probability distribution on the input alphabet to a distribution on the output. In other words, given an input source there is a unique output source determined by the channel. Let us note that in case of abelian algebras every positive map is guaranteed to be completely positive [Tak02]. This is no longer true in the non-abelian case. Hence for the quantum case completely positivity has to be explicitly imposed on (quantum) channels.

We characterize a channel by input/output algebras (of observables) and a positive map. Like the source output we now define a useful quantity called channel output. Corresponding to the atomic basis $\{y_{i}\}$ of $Y$ let $\otimes^{k}y_{i(k)}$ be an atomic basis in $\otimes^{n}Y$. Here $i(k)=(i_{1}i_{2}\dotsc i_{k})$ is a multi-index. Similarly we have an atomic basis $\{\otimes^{k}x_{j(k)}\}$ for $\otimes^{k}X$. The level-$k$ channel output is defined to be.

[TABLE]

Here $C^{(k)}$ represents the channel transition probability matrix on the $k$-fold tensor product corresponding to strings of length $k$. In the DMS case it is simply the $k$-fold tensor product of the matrix $C$. The channel output defined here encodes most important features of the communication process. First, given the input source function 101010We called this the source output before. But as the channel has two terminals we call it input source function to avoid confusion. ${\mathcal{I}}_{\omega^{k}}=\sum_{i}\omega^{k}(x_{i(k)})x_{i(k)}$ the output source function is defined by

[TABLE]

Here, the state $\tilde{\omega}^{k}$ on the output space $\otimes^{k}Y$ can be obtained via the dual $\tilde{\omega}^{k}(y)=\tilde{C}^{k}(\omega^{k})(y)=\omega^{k}(C^{k}(y))$. The formula above is an alternative representation which is very similar to the quantum case. The joint output of the channel can be considered as the combined output of the two terminals of the channel. This is obtained by not tracing out over the input in the equation 9. Thus the joint output

[TABLE]

Let us analyze the algebraic definition of channel given above. For simplicity of notation, we restrict ourselves to level 1. The explicit representation of channel output is

[TABLE]

We interpreted this as follows: if on the channel out-terminal $y_{i}$ is observed then the input could be $x_{j}$ with probability $C(y_{i}|x_{j})\omega(x_{j})/\sum_{j}C(y_{i}|x_{j})\omega(x_{j})$. Now suppose that for a fixed $i$ $C(y_{i}|x_{j})=0$ for all $j$ except one say, $j_{i}$. Then on observing $y_{i}$ at the output we are certain that the the input is $x_{j_{i}}$. If this is true for all values of $y$ then we have an instance of a lossless channel. It is easy to write the channel matrix in this case. Thus, given $1\leq j\leq n$ let $d_{j}$ be the set of integers $i$ for which $C(y_{i}|x_{j})>0$. The lossless property implies that $\{d_{j}\}$ form a partition of the set $\{1,\dotsc,m\}$. The corresponding channel output is

[TABLE]

Clearly lossless channels are the most useful for communication of information. At the other extreme is the useless channel in which there is no correlation between the input and the output. To define it formally, consider a channel ${\mathcal{C}}=\{X,Y,C\}$ as above. The map $C$ induces a map $C^{\prime}:Y\otimes X\rightarrow X$ defined by $C^{\prime}(y\otimes x)=xC(y)$. Given a state $\omega$ on $X$ the dual of the map $C^{\prime}$ defines a state $\Omega_{C}$ on $Y\otimes X$: $\Omega_{C}(y\otimes x)=\omega(C^{\prime}(y\otimes x))=C(y|x)\omega(x)$. We call $\Omega_{C}$ the joint (input-output) state of the channel. A channel is useless if $Y$ and $X$ (identified as $Y\otimes\mathbbm{1}$ and $\mathbbm{1}\otimes X$ resp.) are $\Omega_{C}$-independent.

Lemma 4**.**

A channel ${\mathcal{C}}=\{X,Y,C\}$ with input source $(X,\omega)$ is useless iff the matrix $C_{ij}=C(y_{j}|x_{i})$ is of rank 1.

Proof.

Suppose ${\mathcal{C}}$ is useless. Note that $\Omega_{C}(\mathbbm{1}\otimes x)=\omega(x)$ and $\Omega_{C}(y\otimes\mathbbm{1})=\tilde{\omega}(y)$ where $\tilde{\omega}(y)=\omega(C(y))$ is the image of $\omega$ under the dual of the map $C$. Then $\Omega_{C}$ independence implies $C(y_{j}|x_{i})\omega(x_{i})=\omega(x_{i})\tilde{\omega}(y_{j})$. We may assume that all $\omega(x_{i})>0$ (otherwise we just discard it). Hence, $C(y_{j}|x_{i})=\tilde{\omega}(y_{j})$ and this proves necessity. Now if $C_{ij}$ has rank 1 then all the rows are non-zero multiples of any one row, say the first. Since $C$ is a row stochastic matrix the rows must be identical, that is, $C_{ij}=a_{j}=\tilde{\omega}(y_{j})$ and independence is trivially verified. ∎

The definition of a useless channel captures the intuition that if there is no correlation between the input and output then we can recover practically nothing. The channel coding theorem asserts that apart from this extreme case we can decode the output to recover a large portion of the input with high probability of success. The algebraic version of the channel coding theorem assures that it is possible to approximate, in the long run, an arbitrary channel (excepting the useless case) by a lossless one.

Theorem 7** (Channel coding).**

Let ${\mathcal{C}}$ be a channel with input algebra $X$ and output algebra $Y$. Let $\{x_{i}\}_{i=1}^{n}\text{ and }\{y_{j}\}_{j=1}^{m}$ be atomic bases for $X$ and $Y$ resp. Given a state $\omega$ on $X$, if the channel is not useless then for each $k$ there are subalgebras $Y_{k}\subset\otimes^{k}Y,X_{k}\subset\otimes^{k}X$, a map $C_{k}:Y_{k}\rightarrow X_{k}$ induced by $C$ and a lossless channel $L_{k}:Y_{k}\rightarrow X_{k}$ such that

[TABLE]

Here $\Omega=\otimes^{\infty}\Omega_{C}$ and on $\otimes^{k}Y\otimes\otimes^{k}Y$ it acts as $\Omega^{k}=\otimes^{k}\Omega_{C}$ where $\Omega_{C}$ is the state induced by the channel and a given input state $\omega$. Moreover, if $r_{k}=\text{dim}(X_{k})$ then $R=\frac{\log{r_{k}}}{k}$, called transmission rate, is independent of $k$.

First let us clarify the meaning of the above statements. The theorem simply states that on the chosen set of codewords the channel output of $C_{k}$ induced by the given channel can be made arbitrarily close to that of a lossless channel $L_{k}$. Since a lossless channel has a definite decision scheme for decoding the choice of $L_{k}$ is effectively a decision scheme for decoding the original channel’s output when the input is restricted to our “code-book”. This in turn implies that the probability of error tends to 0.

Proof.

From an atomic basis of $\otimes^{k}X$ choose a subset $A_{k}$ of cardinality $r_{k}$ (to be determined). Let $X_{k}$ be the subalgebra generated by $A_{k}$. Write $C^{(k)}$ for the $k$-fold tensor product of $C$. Let $Q_{k}$ be the identity on $X_{k}$ (it is the sum of all the members of $A_{k}$). For an atomic basis $B_{k}$ of $\otimes^{k}Y$ let $B^{\prime}_{k}$ be the subset such that $C^{(k)}(y)Q_{k}\neq 0\text{ for }y\in B^{\prime}_{k}$. Let $Y_{k}$ be the subalgebra generated by $B^{\prime}_{k}$ and $C_{k}:Y_{k}\rightarrow X_{k}$ denote the linear map $C_{k}(y)=Q_{k}C^{(k)}(y)$. Informally, if we restrict the messages to observables in $A_{k}$ then the output algebra is $Y_{k}$. The new channel map is $C_{k}$. We now have a new channel $\tilde{{\mathcal{C}}}^{k}=(X_{k},Y_{k},C_{k})$. Throughout the rest of the proof we will assume that we are working in $T_{k}$ with the appropriate maps. We next define $L_{k}$ as follows. For $y_{i}\in B^{\prime}_{k}$ let $C_{k}(y_{i})=\sum_{j}C_{k}(y_{i}|x_{j})x_{j},\>x_{j}\in A_{k}$. Let $C_{k}(y_{i}|x_{i_{r}})$ be the maximum of $C_{k}(y_{i}|x_{j})$ for fixed $y_{i}$ (if there are more than one index equal to this maximum choose one arbitrarily). Let $L_{k}(y_{i})=\tilde{\omega}(y_{i})x_{i_{r}}$. The map $L_{k}$ is not unital. Strictly speaking $L_{k}$ is not a channel map as we have defined above. However, as we see below, $L_{k}$ does approximate $O_{C_{k}}$ in $T_{k}$ with small error. What this means is that with high probability we can correctly associate a unique and correct input to a given channel output 111111We have combined two types of decoding scheme: the ideal observer decoding [Ash90] and typical set decoding [CT99]. The non-unital property of $L_{k}$ is reflective of the situation in which some of the original messages outside of $X_{k}$ may end up in $Y_{k}$. Set $r_{k}=2^{kR}$ and let

[TABLE]

Here $O_{\tilde{\omega}^{k}}$ and $O_{\omega^{k}}$ are respectively the input and output source function for the channel $\tilde{{\mathcal{C}}}^{k}$. Let $Z_{k}$ be the identity on the ideal generated by $(\log{O_{C_{k}}}-\log{O_{\tilde{\omega}^{k}}}-k(R+\epsilon))_{+}=(\log{(O_{C_{k}}O_{\tilde{\omega}^{k}}^{-1})}-k(R+\epsilon))_{+},\>\epsilon>0$ in $T_{k}$. 121212This ideal is $T_{k}(\log{(O_{C_{k}}O_{\tilde{\omega}^{k}}^{-1})}-k(R+\epsilon))_{+}$. Note that we write the scalar $k(R+\epsilon)$ instead of the more accurate $k(R+\epsilon)\mathbbm{1}_{k}$ where $\mathbbm{1}_{k}$ is the unit in $T_{k}$.. Note that $O_{C_{k}}=O_{\Omega^{k}}O_{\omega^{k}}^{-1}$ on $T_{k}$. Since

[TABLE]

and $Z_{k}^{2}=Z_{k}$ we conclude that $Z_{k}O_{\omega^{k}}\leq Z_{k}O_{\Omega^{k}}O_{\tilde{\omega}^{k}}^{-1}2^{-k(R+\epsilon)}\leq Z_{k}2^{-k(R+\epsilon)}$. The last inequality follows from the fact that $O_{\Omega^{k}}O_{\tilde{\omega}^{k}}^{-1}\leq\mathbbm{1}$. We also have $O_{L_{k}}\leq O_{C_{k}}$ and $\Omega^{k}(Z_{k})={\tt Tr}(Z_{k}O_{\Omega^{k}})$. The last fact is true for any projection as can be verified using an atomic basis. We now have

[TABLE]

Hence $\Omega(Z_{k}|O_{C_{k}}-O_{L_{k}}|)=\Omega^{k}(Z_{k}(O_{C_{k}}-O_{L_{k}}))\rightarrow 0$ as $k\rightarrow\infty$. To complete the proof we look at the complementary part: $(\mathbbm{1}_{k}-Z_{k})|O_{C_{k}}-O_{L_{k}}|$ where $\mathbbm{1}_{k}$ is the identity in $T_{k}$. Consider the projection $\mathbbm{1}_{k}-Z_{k}$. $Z_{k}$ is the identity in the annihilating ideal of $F_{k-}$ where $F_{k}=(\log{O_{C_{k}}}-\log{O_{\tilde{\omega}^{k}}}-k(R+\epsilon))$. Let $G_{k}=(\log{(\otimes^{k}O_{C}O_{\tilde{\omega}}^{-1})}-k(R+\epsilon)\mathbbm{1})$. Then since $F_{k}$ is the restriction of $G_{k}$ to a subspace $G_{k}=F_{k}+F^{\prime}_{k}$ there is an $F_{k}^{\prime}\in\otimes^{k}Y\otimes\otimes^{k}X$ with $F_{k}F^{\prime}_{k}=0$ (we use the fact the channel is memoryless). Hence taking an approximating polynomial sequence $G_{k-}=F_{k-}+F^{\prime}_{k-}$. It follows that $F_{k-}\leq G_{k-}$ and $Z^{\prime}_{k}$, the identity on the annihilating ideal of $G_{k-}$ satisfies $Z^{\prime}_{k}\leq Z_{k}$. This implies $\Omega(\mathbbm{1}_{k}-Z_{k})\leq\Omega(\mathbbm{1}-Z^{\prime}_{k})$. By definition $\Omega(\mathbbm{1}-Z_{k})=P(\log{\otimes^{k}O_{C}O^{-1}_{\tilde{\omega}}}/k-(R+\epsilon)<0)$ is the probability that $G_{k}<R+\epsilon$. But

[TABLE]

follows from the law of large numbers (see Theorem 5 and its corollary). The quantity $I(X,Y)=\Omega(\log{O_{C}}-\log{O_{\tilde{\omega}}})=H(Y)-H(Y|X)$ is defined as the mutual information between the input and output algebras and $H(Y|X)$ is the conditional entropy. Thus if we have $R<I(X,Y)$, say $R\leq I(X,Y)-2\epsilon$ then $\Omega(\mathbbm{1}-Z^{\prime}_{k})=P(\log{\otimes^{k}O_{C}O^{-1}_{\tilde{\omega}^{k}}}/k-(R+\epsilon)<0)\leq P(|\log{(\otimes^{k}O_{C}O^{-1}_{\tilde{\omega}})}/k-\mathbbm{1}|>\epsilon)$ but the latter $\rightarrow 0$. Putting it all together we have for any $\epsilon>0$ and $R<I-2\epsilon$

[TABLE]

As we already have $\Omega(Z_{k}|O_{C_{k}}-O_{L_{k}}|)\rightarrow 0$ the proof is complete. ∎

The channel coding theorem implies that it is possible to choose a set of “codewords” which can be transmitted with high reliability. It is easy to see that for a lossless channel the input entropy $H(X)$ is equal to the mutual information. We may think of this as conservation of entropy or information which justifies the term “lossless”. Since it is always the case that $H(X)-H(X|Y)=I(X,Y)$ the quantity $H(X|Y)$ can be considered the loss due to the channel. The channel coding theorem is perhaps the most celebrated theorem in Shannon’s work although his proof was not rigorous. The algebraic version of the theorem serves two primary purposes. First, we attempt to make the proof as “algebraic” as possible. More importantly, it gives us the commutative perspective from which we will seek possible extensions to the non-commutative case. Secondly, the channel map $L$ can be used for a decoding scheme. Thus we may think of a coding-decoding scheme for a given channel as a sequence of pairs $(X_{k},L_{k})$ as above.

The coding theorems can be extended to more complicated scenarios like ergodic sources and channels with finite memory. The converse of the channel coding theorem—roughly, any such coding scheme with error tending to 0 (convergence in probability) must have the rate $\log{r_{k}}/k\leq I$—is also true. We will not pursue these issues further here. But we are confident that these generalizations can be appropriately formulated and proved in the algebraic framework.

5 Conclusion and preview of the future work

In the preceding sections we have laid the basic algebraic framework for information theory. This work was devoted to classical parts of information theory corresponding to abelian algebras. Since information theory relies heavily on probabilistic concepts we devoted a major part of the paper to algebraic probability theory. Although, we often confined our discussion to finite-dimensional algebras corresponding to finite sample spaces it is possible to extend it to infinite-dimensional algebras of continuous sample spaces. In this regard, a natural question is: can the algebraic formulation replace Kolmogorov axiomatics based on measure theory? Naively, the answer is no because the assumption of a norm-compete algebra imposes the restriction that the random variables that they represent must be bounded. Moreover, the GNS construction implies that the algebraic framework is essentially equivalent to (almost) bounded random variables on a locally compact space. In order to deal with the unbounded case we have to go beyond the normed algebra structures. A possible course of action is indicated in the examples given in section 3.3: via the use of a “cut-off”. A more general approach would be to consider sequences which converge in a topology weaker than the norm topology to elements of a larger algebra. These and other related issues on foundations are deep and merit a separate investigation.

The second major theme of this paper is information theory in the algebraic framework. As some the most important results of information theory concern finite or discrete alphabet we have primarily dealt with these cases only. In this context, we can treat ergodic sources, channels with finite memory and multi-terminal channels. These topics will be investigated in the future in the non-commutative setting. However, let us recall one of the principal motivation of this paper: the construction of a single framework for dealing with quantum and classical information. We have seen that the algebraic theory in the commutative case already indicates the close analogies between the two cases. We will delve deeper into these analogies and aim to throw light on some basic issues like quantum Huffman coding [BFGL00], channel capacities and general no-go theorems among others, once we formulate the appropriate models. In this context, let us mention that many investigators have recognized the importance of the algebraic framework but a comprehensive algebraic model which can be extended to infinite-dimensional case is lacking. We aim to address these important issues in subsequent work.