Distributed compression through the lens of algorithmic information theory: a primer

TL;DR

This paper introduces an accessible overview of a new approach to distributed data compression based on Algorithmic Information Theory, which applies to individual data strings without assuming a probabilistic model.

Contribution

It presents a simplified, accessible explanation of an AIT-based analogue of the Slepian-Wolf theorem that works for individual data strings, not just stochastic models.

Findings

The AIT version of the Slepian-Wolf theorem applies to individual strings.

It requires knowledge of the complexity profile of input data.

The approach does not rely on probabilistic assumptions.

Abstract

Distributed compression is the task of compressing correlated data by several parties, each one possessing one piece of data and acting separately. The classical Slepian-Wolf theorem (D. Slepian, J. K. Wolf, IEEE Transactions on Inf. Theory, 1973) shows that if data is generated by independent draws from a joint distribution, that is by a memoryless stochastic process, then distributed compression can achieve the same compression rates as centralized compression when the parties act together. Recently, the author (M. Zimand, STOC 2017) has obtained an analogue version of the Slepian-Wolf theorem in the framework of Algorithmic Information Theory (also known as Kolmogorov complexity). The advantage over the classical theorem, is that the AIT version works for individual strings, without any assumption regarding the generative process. The only requirement is that the parties know the complexity profile of the input strings, which is a simple quantitative measure of the data correlation. The goal of this paper is to present in an accessible form that omits some technical details the main ideas from the reference (M. Zimand, STOC 2017).

Equations

$$n_A+n_B+n_C\ge 5n,$$

$$n_A+n_B\ge 3n,n_A+n_C\ge 3n,n_B+n_C\ge 3n,$$

$$n_A\ge n,n_B\ge n,n_C\ge n,$$

$$H(X)=\sum _{x\in \Sigma }P(X=x)(1/\log P(X=x)).$$

$$x_1=00000000000000000000000000000000$$

$$x_2=10110000010101110101010011011100$$

$$C_{M}(x)=\min\{|p|\mid\mbox{ $p$ is a program for $x$ relative to $M$}\}.$$

$$C_U(x)\le C_M(x)+|m|.$$

$$D(E_{A}(x_{A}),E_{B}(x_{B}),E_{C}(x_{C}))=(x_{A},x_{B},x_{C}),\mbox{~{}~{}with probability $1-\epsilon$},$$

$$\begin{array}[]{rl}|E_{A}(x_{A})|+|E_{B}(x_{B})|+|E_{C}(x_{C})|&\geq C(x_{A},x_{B},x_{C})+\log(1-\epsilon)-O(1),\\ |E_{A}(x_{A})|+|E_{B}(x_{B})|&\geq C(x_{A},x_{B}\mid x_{C})+\log(1-\epsilon)-O(1),\\ |E_{A}(x_{A})|&\geq C(x_{A}\mid x_{B},x_{C})+\log(1-\epsilon)-O(1),\\ \end{array}$$

$$\sum_{i\in V}|E_{i}(x_{i})|\geq C(x_{V}\mid x_{\{A,B,C\}-V})+\log(1-\epsilon)-O(1),\mbox{ for all $V\subseteq\{A,B,C\}$},$$

$$\sum_{i\in V}n_{i}\geq C(x_{V}\mid x_{\{A,B,C\}-V}),\mbox{ for all $V\subseteq\{A,B,C\}$},$$

$$B_1=\{x\in \{0,1{\}}^n\mid C(x)\le C(x_A)\}.$$

$$B_2=\{x\in \{0,1{\}}^n\mid C(x\mid p_A)\le C(x_B\mid p_A)\}.$$

$$B_1^{\prime }=\{x\in \{0,1{\}}^n\mid C(x\mid x_B)\le C(x_A\mid x_B)\}.$$

$$B_2^{\prime }=\{x\in \{0,1{\}}^n\mid C(x\mid p_A)\le n_B\},$$

$$\begin{array}[]{ll}C(x_{B}\mid x_{C},p_{A})&=^{+}C(x_{B},x_{C},p_{A})-C(x_{C},p_{A})\\ &\leq^{+}C(x_{A},x_{B},x_{C})-C(x_{C},p_{A})\\ &\leq^{+}(n_{A}+n_{B}+n_{C})-(n_{A}+n_{C})=n_{B}.\end{array}$$

$$\begin{array}[]{ll}C(x_{B},x_{C}\mid p_{A})&=^{+}C(x_{B},x_{C},p_{A})-C(p_{A})\\ &\leq^{+}C(x_{A},x_{B},x_{C})-C(p_{A})\\ &\leq^{+}(n_{A}+n_{B}+n_{C})-n_{A}=n_{B}+n_{C}.\end{array}$$

$$\bigg{|}\frac{|E(B,A)|}{|B|\cdot D}-\frac{|A|}{|R|}\bigg{|}\leq\epsilon,$$

$$\rm{Prob}\bigg{[}\bigg{|}\frac{|E^{\prime}(B,A)|}{|B|\cdot D}-\frac{|A|}{|R^{\prime}|}\bigg{|}>\epsilon\bigg{]}\leq 2^{-\Omega(K^{\prime}\cdot D\cdot\epsilon^{2})}.$$

Full text

1 On busy friends wishing to share points

Zack has three good friends, Alice, Bob, and Charles, who share with him every piece of information they have. One day, Alice, Bob, and Charles, separately, observe three collinear points $A$, respectively $B$ and $C$, in the $2$-dimensional affine space over the field with $2^{n}$ elements. Thus, each one of Alice, Bob, and Charles possesses $2n$ bits of information, giving the two coordinates of their respective points. Due to the geometric relation, collectively, they have $5n$ bits of information, because given two points the third one can be described with just one coordinate. They want to email the points to Zack, without wasting bandwidth, that is by sending approximately $5n$ bits, where “approximately” means that they can afford an overhead of $O(\log n)$ bits. Clearly, if they collaborate, they can send exactly $5n$ bits. The problem is that they have busy schedules, and cannot find a good time to get together and thus they have to compress their points in isolation. How many bits do they need to send to Zack? Let us first note some necessary requirements for the compression lengths. Let $n_{A}$ be the number of bits to which Alice compresses her point $A$, and let $n_{B}$ and $n_{C}$ have the analogous meaning for Bob and Charles. It is necessary that

[TABLE]

because Zack needs to acquire $5n$ bits. It is also necessary that

[TABLE]

because if Zack gets somehow one of the three points, he still needs $3n$ bits of informations from the other two points. And it is also necessary that

[TABLE]

because if Zack gets somehow two of the three points, he still needs $n$ bits of informations from the remaining point.

We will see that any numbers $n_{A},n_{B}$ and $n_{C}$ satisfying the above necessary conditions, are also sufficient up to a small logarithmic overhead, in the sense that there are probabilistic compression algorithms such that if $n_{A},n_{B}$ and $n_{C}$ satisfy these conditions, then Alice can compress point $A$ to a binary string $p_{A}$ of length $n_{A}+O(\log n)$, Bob can compress point $B$ to a binary string $p_{B}$ of length $n_{B}+O(\log n)$, Charles can compress point $C$ to a binary string $p_{C}$ of length $n_{C}+O(\log n)$, and Zack can with high probability reconstruct the three points from $p_{A},p_{B}$ and $p_{C}$. Moreover, the compression does not use the geometric relation between the points, but only the correlation of information in the points, as expressed in the very flexible framework of algorithmic information theory.

2 Algorithmic information theory

Algorithmic Information Theory (AIT), initiated independently by Solomonoff [Sol64], Kolmogorov [Kol65], and Chaitin [Cha66], is a counterpart to the Information Theory (IT), initiated by Shannon. In IT the central object is a random variable $X$ whose realizations are strings over an alphabet $\Sigma$. The Shannon entropy of $X$ is defined by

[TABLE]

The entropy $H(X)$ is viewed as the amount of information in $X$, because each string $x$ can be described with $\lceil 1/\log P(X=x)\rceil$ bits (using the Shannon code), and therefore $H(X)$ is the expected number of bits needed to describe the outcome of the random process modeled by $X$.

AIT dispenses with the stochastic generative model, and defines the complexity of an individual string $x$ as the length of its shortest description. For example, the string

[TABLE]

has low complexity because it can be succinctly described as “$2^{5}$ zeros.” The string

[TABLE]

is a $32$-bit string obtained using random atmospheric noise (according to random.org), and has high complexity because it does not have a short description.

Formally, given a Turing machine $M$, a string $p$ is said to be a program (or a description) of a string $x$, if $M$ on input $p$ prints $x$. We denote the length of a binary string $x$ by $|x|$. The Kolmogorov complexity of $x$ relative to the Turing machine $M$ is

[TABLE]

If $U$ is universal Turing machine, then for every other Turing machine $M$ there exists a string $m$ such that $U(m,p)=M(p)$ for all $p$, and therefore for every string $x$,

[TABLE]

Thus, if we ignore the additive constant $|m|$, the Kolmogorov complexity of $x$ relative to $U$ is minimal. We fix a universal Turing machine $U$, drop the subscript $U$ in $C_{U}(\cdot)$, and denote the complexity of $x$ by $C(x)$. We list below a few basic facts about Kolmogorov complexity:

For every string $x$, $C(x)\leq|x|+O(1)$, because a string $x$ is trivially described by itself. (Formally, there is a Turing machine $M$ that, for every $x$, on input $x$ prints $x$.) 2. 2.

Similarly to the complexity of $x$, we define the complexity of $x$ conditioned by $y$ as C(x\mid y)=\min\{|p|\mid\mbox{ U$on input$p$and$y$prints$x}\}. 3. 3.

Using some standard pairing function $\langle\cdot,\cdot\rangle$ that maps pair of strings into single strings, we define $C(x,y)$ the complexity of a pair of strings (and then we can extend to tuples with larger arity) by $C(x,y)=C(<x,y>)$. 4. 4.

We use the convenient shorthand notation $a\leq^{+}b$ to mean that $a\leq b+O(\log(a+b))$, where the constant hidden in the $O(\cdot)$ notation only depends on the universal machine $U$. Similarly $a\geq^{+}b$ means $a\geq b-O(\log(a+b))$, and $a=^{+}b$ means ($a\leq^{+}b$ and $a\geq^{+}b$). 5. 5.

The chain rule in information theory states that $H(X,Y)=H(X)+H(Y\mid X)$. A similar rule holds true in algorithmic information theory: for all $x$ and $y$, $C(x,y)=^{+}C(x)+C(y\mid x)$.

3 Distributed compression, more formally

We present the problem confronting Alice, Bob, Charles (the senders) and Zack (the receiver) in an abstract and formal setting. We assume that each one of Alice, Bob, and Charles has $n$ bits of information, which, in concrete terms, means that Alice has an $n$-bit binary string $x_{A}$, Bob has an $n$-bit binary string $x_{B}$, and Charles has an $n$-bit binary string $x_{C}$. We also assume that the $3$-tuplet $(x_{A},x_{B},x_{C})$ belongs to a set $S\subseteq\{0,1\}^{n}\times\{0,1\}^{n}\times\{0,1\}^{n}$, which defines the way in which the information is correlated (for example, $S$ may be the set of all three collinear points) and that all parties (i.e., Alice, Bob, Charles, and Zack) know $S$. Alice is using an encoding fumction $E_{A}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{A}}$, Bob is using an encoding fumction $E_{B}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{B}}$, Charles is using an encoding fumction $E_{C}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{C}}$, and Zack is using a decoding fumction $D:\{0,1\}^{n_{A}}\times\{0,1\}^{n_{B}}\times\{0,1\}^{n_{C}}\rightarrow\{0,1\}^{n}$. Ideally, the requirement is that for all $(x_{A},x_{B},x_{C})$ in $S$, $D(E_{A}(x_{A}),E_{B}(x_{B}),E_{C}(x_{C}))=(x_{A},x_{B},x_{C})$. However, since typically the encoding functions are probabilistic, we allow the above equality to fail with probability bounded by some small $\epsilon$, where the probability is over the random bits used by the encoding functions. Also, sometimes, we will be content if the encoding/decoding procedures work, not for all, but only for “most” $3$-tuples in $S$ (i.e., with probability close to $1$, under a given probability distribution on $S$).

Our focus in this paper is to present distributed compression in the framework of Algorithmic Information Theory, but let us present first the point of view of Information Theory, where the problem has been studied early on. The celebrated classical theorem of Slepian and Wolf [SW73] characterizes the possible compression rates $n_{A},n_{B}$ and $n_{C}$ for the case of memoryless sources. The memoryless assumption means that $(x_{A},x_{B},x_{C})$ are realizations of random variables $(X_{A},X_{B},X_{C})$, which consist of $n$ independent copies of a random variable that has a joint distribution $P(b_{1},b_{2},b_{3})$ on triples of bits. In other words, the generative model for $(x_{A},x_{B},x_{C})$ is a stochastic process that consists of $n$ independent draws from the joint distribution, such that Alice observes $x_{A}$, the sequence of first components in the $n$ draws, Bob observes $x_{B}$, the second components, and Charles observes $x_{C}$, the third components. A stochastic process of this type is called 3-DMS (Discrete Memoryless Source). By Shannon’s Source Coding Theorem, if $n^{\prime}$ is a number that is at least $H(X_{A},X_{B},X_{C})$ and if Alice, Bob, and Charles put their data together, then, for every $\epsilon>0$, there exists an encoding/decoding pair $E$ and $D$, where $E$ compresses $3n$-bit strings to $(n^{\prime}+\epsilon n)$-bit strings and $D(E(X_{A},X_{B},X_{C}))=(X_{A},X_{B},X_{C})$ with probability $1-\epsilon$, provided $n$ is large enough. The second part of Shannon’s Source Coding Theorem shows that this is essentially optimal because if the data is compressed to length smaller than $H(X_{A},X_{B},X_{C})-\epsilon n$ (for constant $\epsilon$), then the probability of correct decoding goes to [math]. The Slepian-Wolf Theorem shows that such a compression can also be done if Alice, Bob, and Charles compress separately. Actually, it describes precisely the possible compression lengths. Note that if the three senders compress separately to lengths $n_{A},n_{B}$ and $n_{C}$ as indicated above, then it is essentially necessary that $n_{A}+n_{B}+n_{C}\geq H(X_{A},X_{B},X_{C})-\epsilon n$, $n_{A}+n_{B}\geq H(X_{A},X_{B}\mid X_{C})-\epsilon n$ (because even if Zack has $X_{C}$, he still needs to receive a number of bits equal to the amount of entropy in $X_{A}$ and $X_{B}$ conditioned by $X_{C}$), $n_{A}\geq H(X_{A}\mid X_{B},X_{C})-\epsilon n$ (similarly, even if Zack has $X_{B}$ and $X_{C}$, he still needs to receive a number of bits equal to the amount of entropy in $X_{A}$ conditioned by $X_{B}$ and $X_{C}$), and there are the obvious other necessary conditions obtained by permuting $A,B$ and $C$. The Slepian-Wolf Theorem shows that for 3-DMS these necessary conditions are, essentially, also sufficient, in the sense that the slight change of $-\epsilon n$ into $+\epsilon n$ allows encoding/decoding procedures. Thus, in the next theorem we suppose that $n_{A}+n_{B}+n_{C}\geq H(X_{A},X_{B},X_{C})+\epsilon n$, and similarly for the other relations.

Theorem 3.1** (Slepian-Wolf Theorem [SW73]).**

Let $(X_{A},X_{B},X_{C})$ be a 3-DMS, let $\epsilon>0$, and let $n_{A},n_{B},n_{C}$ satisfy the above conditions (with $+\epsilon n$ instead of $-\epsilon n$). Then there exist encoding functions $E_{A}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{A}},E_{B}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{B}},E_{C}:\{0,1\}^{n}\rightarrow\{0,1\}^{n_{C}}$ and a decoding function $D:\{0,1\}^{n_{A}}\times\{0,1\}^{n_{B}}\times\{0,1\}^{n_{C}}\rightarrow\{0,1\}^{n}$ such that $Prob[D(E(X_{A},X_{B},X_{C}))=(X_{A},X_{B},X_{C})]\geq 1-O(\epsilon)$, provided $n$ is large enough.

There is nothing special about three senders, and indeed the Slepian-Wolf theorem holds for any number $\ell$ of senders, where $\ell$ is a constant, and for sources which are $\ell$-DMS over any alphabet $\Sigma$. This means that the senders compress $(x_{1},\ldots,x_{\ell})$, which is realization of random variables $(X_{1},\ldots,X_{\ell})$, obtained from $n$ independent draws from a joint distribution $p(a_{1},\ldots,a_{\ell})$, with each $a_{i}$ ranging over the alphabet $\Sigma$. The $i$-th sender observes the realization $x_{i}$ of $X_{i}$, and uses an encoding function $E_{i}:\Sigma^{n}\rightarrow\Sigma^{n_{i}}$. Suppose that the compression lengths $n_{i}$, $i=1,\ldots,\ell$, satisfy $\sum_{i\in V}n_{i}\geq H(X_{V}\mid X_{\overline{V}})+\epsilon n$, for every subset $V\subseteq\{1,\ldots,\ell\}$ (where if $V=\{i_{1},\ldots,i_{t}\}$, $X_{V}$ denotes the tuple $(X_{i_{1}},\ldots,X_{i_{t}})$, and $\overline{V}$ denotes $\{1,\ldots,\ell\}-V$). Then the Slepian-Wolf theorem for $\ell$-DMS states that there are $E_{1},\ldots,E_{\ell}$ of the above type, and $D:\Sigma^{n_{1}}\times\ldots\times\Sigma^{n_{\ell}}\rightarrow\Sigma^{n}$ such that $D(E_{1}(X_{1}),\ldots,E_{\ell}(X_{\ell}))=(X_{1},\ldots,X_{\ell})$ with probability $1-\epsilon$.

As pointed out above, the Slepian-Wolf theorem shows the surprising and remarkable fact that, for memoryless sources, distributed compression can be done at an optimality level that is on a par with centralized compression. On the weak side, the memoryless property means that there is a lot of independence in the generative process: the realization at time $i$ is independent of the realization at time $i-1$. Intuitively, independence helps distributed compression. For example, in the limit case in which the senders observe realizations of fully independent random variables, then, clearly, it makes no difference whether compression is distributed or centralized. The Slepian-Wolf theorem has been extended to sources that are stationary and ergodic [Cov75], but these sources are still quite simple, and intuitively realizations which are temporally sufficiently apart are close to being independent.

One may be inclined to believe that the optimal compression rates of distributed compression in the theorem are caused by the independence properties of the sources. However, this is not so, and we shall see that in fact the Slepian-Wolf phenomenon does not require any type of independence. Even more, it works without any generative model. For that we need to work in the framework of Kolmogorov complexity (AIT).

Let us recall the example from Section 1: Alice, Bob, and Charles observe separately, respectively, the collinear points $A,B,C$. Even without assuming any generative process for the three points, we can still express their correlation using Kolmogorov complexity. More precisely, their correlation is described by the Kolmogorov complexity profile, which consists of $7$ numbers, giving the complexities of all non-empty subsets of $\{A,B,C\}$: $(C(A),C(B),C(C),C(A,B),C(A,C),C(B,C),C(A,B,C))$.

Let us consider the general case, in which the three senders have, respectively, $n$-bit strings $x_{A},x_{B},x_{C}$ having a given complexity profile $(C(x_{V})\mid V\subseteq\{x_{A},x_{B},x_{C}\}$ , $V\not=\emptyset)$ (where $x_{V}$ is the notation convention that we used for the $\ell$-senders case of the Slepian-Wolf theorem). What are the possible compression lengths, so that Zack can decompress and obtain $(x_{A},x_{B},x_{C})$ with probability $(1-\epsilon)$?

To answer this question, for simplicity, let us consider the case of a single sender, Alice. She wants to use a probabilistic encoding function $E$ such that there exists a decoding function $D$ with the property that for all $n$, and for all $n$-bit strings $x$, $D(E(x))=x$, with probability $1-\epsilon$. A lower bound on the length $|E(x)|$ is given in the following lemma.

Lemma 3.2**.**

Let $E$ be a probabilistic encoding function, and $D$ be a decoding function such that for all strings $x$, $D(E(x))=x$, with probability $(1-\epsilon)$. Then for every $k$, there is a string $x$ with $C(x)\leq k$, such that $|E(x)|\geq k+\log(1-\epsilon)-O(1)$.

Proof.

Fix $k$ and let $S=\{x\mid C(x)\leq k\}$. It can be shown that for some constant $c$, $|S|\geq 2^{k-c}$, where $|S|$ is the size of $S$ (the idea is that the first string which is not in $S$ can be described with $\log|S|+O(1)$ bits). Since for every $x\in S$, $D(E(x,\rho))=x$ with probability $1-\epsilon$ over the randomness $\rho$, there is some fixed randomness $\rho$ such that $D(E(x,\rho))=x$, for a fraction of $1-\epsilon$ of the $x$’s in $S$. Let $S^{\prime}\subseteq S$ be the set of such strings $x$. Thus, $|S^{\prime}|\geq(1-\epsilon)|S|\geq(1-\epsilon)2^{k-c}$ and the function $E(\cdot,\rho)$ is one-to-one on $S^{\prime}$ (otherwise decoding would not be possible). Therefore the function $E(\cdot,\rho)$ cannot map all $S^{\prime}$ into strings of length $k+\log(1-\epsilon)-(c+1)$. ∎

In short, if for every $x$, $D(E(x))=x$ with probability $1-\epsilon$, then for infinitely many $x$ it must be the case that $|E(x)|\geq C(x)+\log(1-\epsilon)-O(1)$. In other words, if we ignore the small terms, it is not possible to compress to length less than $C(x)$.

In the same way, similar lower bounds can be established for the case of more senders. For example, let us consider three senders that use the probabilistic encoding functions $E_{A},E_{B}$ and $E_{C}$: if there is a decoding function $D$ such that for every $(x_{A},x_{B},x_{C})$,

[TABLE]

(where the probability is over the randomness used by the encoding procedures) then for infinitely many $(x_{A},x_{B},x_{C})$

[TABLE]

and similar relations hold for any permutation of $A,B$ and $C$. As we did above, it is convenient to use the notation convention that if $V$ is a subset of $\{A,B,C\}$, we let $x_{V}$ denote the tuple of strings with indices in $V$ (for example, if $V=\{A,C\}$, then $x_{V}=(x_{A},x_{C})$). Then the above relations can be written concisely as

[TABLE]

The next theorem is the focal point of this paper. It shows that the above necessary conditions regarding the compression lengths are, essentially, also sufficient.

Theorem 3.3** (Kolmogorov complexity version of Slepian-Wolf coding [Zim17] ).**

There exist probabilistic algorithms $E_{A},E_{B},E_{C}$, a deterministic algorithm $D$, and a function $\alpha(n)=O(\log n)$ such that for every $n$, for every tuple of integers $(n_{A},n_{B},n_{C})$, and for every tuple of $n$-bit strings $(x_{A},x_{B},x_{C})$ if

[TABLE]

then

• (a)

$E_{A}$* on input $x_{A}$ and $n_{A}$ outputs a string $p_{A}$ of length at most $n_{A}+\alpha(n)$, $E_{B}$ on input $x_{B}$ and $n_{B}$ outputs a string $p_{B}$ of length at most $n_{B}+\alpha(n)$, $E_{C}$ on input $x_{C}$ and $n_{C}$ outputs a string $p_{C}$ of length at most $n_{C}+\alpha(n)$,*

• (b)

$D$* on input $(p_{A},p_{B},p_{C})$ outputs $(x_{A},x_{B},x_{C})$, with probability $1-1/n$.*

We present the proof of this theorem in the next section, but for now, we make several remarks:

• •

Compression procedures for individual inputs (i.e., without using any knowledge regarding the generative process) have been previously designed using the celebrated Lempel-Ziv methods [LZ76, Ziv78]. Such methods have been used for distributed compression as well [Ziv84, DW85, Kuz09]. For such procedures two kinds of optimality have been established, both valid for infinite sequences and thus having an asymptotic nature. First, the procedures achieve a compression length that is asymptotically equal to the so-called finite-state complexity, which is the minimum length that can be achieved by finite-state encoding/decoding procedures. Secondly, the compression rates are asymptotically optimal in case the infinite sequences are generated by sources that are stationary and ergodic [WZ94]. In contrast, the compression in Theorem 3.3 applies to finite strings and achieves a compression length close to minimal description length. On the other hand, the Lempel-Ziv approach has lead to efficient compression algorithms that are used in practice.

• •

At the cost of increasing the “overhead” $\alpha(n)$ from $O(\log n)$ to $O(\log^{3}n)$, we can obtain compression procedures $E_{A},E_{B}$ and $E_{C}$ that run in polynomial time. On the other hand, the decompression procedure $D$ is slower than any computable function. This is unavoidable at this level of optimality (compression at close to minimum description length) because of the existence of deep strings. (Informally, a string $x$ is deep if it has a description $p$ of small length but the universal machine takes a long time to produce $x$ from $p$.)

• •

The theorem is true for any number $\ell$ of senders, where $\ell$ is an arbitrary constant. We have singled out $\ell=3$ because this case allows us to present the main ideas of the proof in a relatively simple form.

• •

Romashchenko [Rom05] (building on an earlier result of Muchnik [Muc02]) has obtained a Kolmogorov complexity version of Slepian-Wolf, in which the encoding and the decoding functions use $O(\log n)$ of extra information, called help bits. The above theorem eliminates the help bits, and is, therefore, fully effective. The cost is that the encoding procedure is probabilistic and thus there is a small error probability. The proof of Theorem 3.3 is inspired from Romashchenko’s approach, but the technical machinery is quite different.

• •

The classical Slepian-Wolf theorem. can be obtained from the Kolmogorov complexity version because if $X$ is memoryless, then with probability $1-\epsilon$, $H(X)-c_{\epsilon}\sqrt{n}\leq C(X)\leq H(X)+c_{\epsilon}\sqrt{n}$, where $c_{\epsilon}$ is a constant that only depends on $\epsilon$.

4 Proof sketch of Theorem 3.3

The central piece in the proof is a certain type of bipartite graph with a low congestion property. We recall that in a bipartite graph, the nodes are partitioned in two sets, $L$ (the left nodes) and $R$ (the right nodes), and all edges connect a left node to a right node. We allow multiple edges between two nodes. In the graphs that we use, all left nodes have the same degree, called the left degree. Specifically, we use bipartite graphs $G$ with $L=\{0,1\}^{n}$, $R=\{0,1\}^{m}$ and with left degree $D=2^{d}$. We label the edges outgoing from $x\in L$ with strings $y\in\{0,1\}^{d}$. We typically work with a family of graphs indexed on $n$ and such a family of graphs is computable if there is an algorithm that on input $(x,y)$, where $x\in L$ and $y\in\{0,1\}^{d}$, outputs the $y$-th neighbor of $x$. Some of the graphs also depend on a rational $0<\delta<1$. A constructible family of graphs is explicit if the above algorithm runs in time $\rm{poly}(n,1/\delta)$.

We now introduce informally the notions of a rich owner and of a graph with the rich owner property. Let $B\subseteq L$. The $B$-degree of a right node is the number of its neighbors that are in $B$. Roughly speaking a left node is a rich owner with respect to $B$, if most of its right neighbors are “well-behaved,” in the sense that their $B$-degree is not much larger than $|B|\cdot D/|R|$, the average right degree when the left side is restricted to $B$. One particularly interesting case, which is used many times in the proof, is when most of the neighbors of a left $x$ have $B$-degree $1$, i.e., when $x$ “owns” most of its right neighbbors. A graph has the rich owner property if, for all $B\subseteq L$, most of the left nodes in $B$ are rich owners with respect to $B$. In the formal definition below, we replace the average right degree with a value which may look arbitrary, but since in applications, this value is approximately equal to the average right degree, the above intuition should be helpful.

The precise definition of rich ownership depends on two parameters $k$ and $\delta$.

Definition 4.1**.**

Let $G$ be a bipartite graph as above and let $B$ be a subset of $L$. We say that $x\in B$ is a $(k,\delta)$-rich owner with respect to $B$ if the following holds:

• •

small regime case:* If $|B|\leq 2^{k}$, then at least $1-\delta$ fraction of $x$’s neighbors have $B$-degree equal to $1$, that is they are not shared with any other nodes in $B$. We also say that $x\in B$ owns $y$ with respect to B if $y$ is a neighbor of $x$ and the $B$-degree of $y$ is $1$.*

• •

large regime case:* If $|B|>2^{k}$, then at least a $1-\delta$ fraction of $x$’s neighbors have $B$-degree at most $(2/\delta^{2})|B|\cdot D/2^{k}$.*

If $x$ is not a $(k,\delta)$-rich owner with respect to $B$, then it is said to be a $(k,\delta)$-poor owner with respect to $B$.

Definition 4.2**.**

A bipartite graph $G=(L=\{0,1\}^{n},R=\{0,1\}^{m},E\subseteq L\times R)$ has the $(k,\delta)$-rich owner property if for every set $B\subseteq L$ all nodes in $B$, except at most $\delta|B|$ of them, are $(k,\delta)$-rich owners with respect to $B$.

The following theorem provides the type of graph that we use.

Theorem 4.3**.**

For every natural numbers $n$ and $k$ and for every rational number $\delta\in(0,1]$, there exists a computable bipartite graph $G=(L,R,E\subseteq L\times R)$ that has the $(k,\delta)$-rich property with the following parameters: $L=\{0,1\}^{n}$, $R=\{0,1\}^{k+\gamma(n/\delta)}$, left degree $D=2^{\gamma(n/\delta)}$, where $\gamma(n)=O(\log n)$.

There also exists an explicit bipartite graph with the same parameters except that the overhead is $\gamma(n)=O(\log^{3}n)$.

The graphs in Theorem 4.3 are derived from randomness extractors. The computable graph is obtained with the probabilistic method, and we sketch the construction in Section 5. The explicit graph relies on the extractor from [RRV99] and uses a combination of techniques from [RR99], [CRVW02], and [BZ14].

Let us proceed now to the proof sketch of Theorem 3.3. We warn the reader that for the sake of readability, we skip several technical elements. In particular, we ignore the loss of precision in $=^{+},\leq^{+},\geq^{+}$, and we treat these relations as if they were $=,\leq,\geq$.

Recall that the input procedures $E_{A},E_{B}$ and $E_{C}$ have as inputs, respectively, the pairs $(x_{A},n_{A}),(x_{B},n_{B}),(x_{C},n_{C})$, where $x_{A},x_{B},x_{C}$ are $n$-bit strings, and $n_{A},n_{B},n_{C}$ are natural numbers. The three encoding procedures use, respectively the graphs $G_{A},G_{B}$ and $G_{C}$, which have, respectively, the $(n_{A}+1,1/n^{2})$, $(n_{B}+1,1/n^{2})$, $n_{C}+1,1/n^{2})$ rich owner property. Viewing the strings $x_{A},x_{B},x_{C}$ as left nodes in the respective graphs, the encoding procedures pick $p_{A},p_{B},p_{C}$ as random neighbors of $x_{A},x_{B},x_{C}$ (see Figure 2).

We need to show that if $n_{A},n_{B},n_{C}$ satisfy the inequalities (1), then it is possible to reconstruct $(x_{A},x_{B},x_{C})$ from $(p_{A},p_{B},p_{C})$ with high probability (over the random choice of $(p_{A},p_{B},p_{C})$). The general idea is to identify computable enumerable subsets $B_{1},B_{2},B_{3}$ of left nodes in the three graphs, which are in the “small regime,” and which contain respectively $x_{A},x_{B},x_{C}$ as rich owners. Then $p_{A}$ has $x_{A}$ as its single neighbor in $B_{1}$, and therefore $x_{A}$ can be obtained from $p_{A}$ by enumerating the elements of $B_{1}$ till we find one that has $p_{A}$ as a neighbor ($x_{B},x_{C}$ are obtained similarly).

We shall assume first that the decoding procedure $D$ knows the $7$-tuple $(C(x_{V})\mid V\subseteq\{A,B,C\},V\not=\emptyset)$, i.e., the complexity profile of $(x_{A},x_{B},x_{C})$.

The proof has an inductive character, so let us begin by analyzing the case when there is a single sender, then when there are two senders, and finally when there are three senders.

1 Sender. We show how to reconstruct $x_{A}$ from $p_{A}$, assuming $C(x_{A})\leq n_{A}$. Let

[TABLE]

Since the size of $B_{1}$ is bounded by $2^{C(x_{A})+1}\leq 2^{n_{A}+1}$, it follows that $B_{1}$ is in the small regime in $G_{A}$. The number of poor owners with respect to $B_{1}$ in $G_{A}$ is at most $(1/n^{2})\cdot 2^{C(x_{A})+1}\approx 2^{C(x_{A})-2\log n}$, and it can be shown that any poor owner can be described by $C(x_{A})-\Omega(\log n)$ bits (essentially by its rank in some fixed standard ordering of the set of poor owners). Therefore the complexity of a poor owner is strictly less than $C(x_{A})$ and thus $x_{A}$ is a rich owner with respect to $B_{1}$, as needed to enable its reconstruction from $p_{A}$.

2 Senders. We show how to reconstruct $x_{A},x_{B}$ from $p_{A},p_{B}$, assuming $C(x_{A}\mid x_{B})\leq n_{A},C(x_{B}\mid x_{A})\leq n_{B},C(x_{A},x_{B})\leq n_{A}+n_{B}$.

If $n_{A}\geq C(x_{A})$, then $x_{A}$ can be reconstructed from $p_{A}$ as in the 1 Sender case. Next, since $n_{B}\geq C(x_{B}\mid x_{A})$, $x_{B}$ can be reconstructed from $x_{A}$ and $p_{B}$, similar to the 1 Sender case.

So let us assume that $C(x_{A})>n_{A}$. Let

[TABLE]

We show below that (1) $B_{2}$ is in the small regime in $G_{B}$, and (2) that it can be effectively enumerated. Since $x_{B}\in B_{2}$, and since it is a rich owner with respect to $B_{2}$ (by a similar argument with the one used for $x_{A}$ in the 1 Sender case), this implies that $x_{B}$ can be reconstructed from $p_{A},p_{B}$, and next $x_{A}$ can be reconstructed from $x_{B}$ and $p_{A}$, as in the 1 Sender case (because $n_{A}\geq C(x_{A}\mid x_{B})$).

It remains to prove the assertions (1) and (2) claimed above. We establish the following fact.

Fact 1**.**

1. (a)

$C(p_{A})=^{+}n_{A}$, 2. (b)

$C(p_{A},x_{B})=^{+}C(x_{A},x_{B})$. 3. (c)

$C(x_{B}\mid p_{A})=^{+}C(x_{A},x_{B})-n_{A}$. 4. (d)

$C(x_{B}\mid p_{A})\leq^{+}n_{B}$.

Proof.

(a) By the same argument used above, $x_{A}$ is still a rich owner with respect to $B_{1}$, but $B_{1}$ is now in the large regime. This implies that with probability $1-(1/n^{2})$, $p_{A}$ has, for some constant $c$, $2^{C(x_{A})-n_{A}+c\log n}$ neighbors in $B_{1}$, one of them being $x_{A}$. The string $x_{A}$ can be constructed from $p_{A}$ and its rank among $p_{A}$’s neighbors in $B_{1}$. This implies $C(x_{A})\leq^{+}C(p_{A})+(C(x_{A})-n_{A})$, and thus, $C(p_{A})\geq^{+}n_{A}$. Since $|p_{A}|\leq^{+}n_{A}$, it follows that $C(p_{A})\leq^{+}n_{A}$, and therefore, $C(p_{A})=^{+}n_{A}$.

(b) The “$\leq^{+}$” inequality holds because $p_{A}$ can be obtained from $x_{A}$ and $O(\log n)$ bits which describe the edge $(x_{A},p_{A})$ in $G_{A}$. For the “$\geq^{+}$” inequality, let

[TABLE]

$B_{1}^{\prime}$ is in the small regime in $G_{A}$ (because $|B_{1}^{\prime}|\leq 2^{C(x_{A}\mid x_{B})+1}\leq 2^{n_{A}+1}$), and $x_{A}$ is a rich owner with respect to $B_{1}^{\prime}$ (because, as we have argued above, poor owners, being few, have complexity conditioned by $x_{B}$, less than $C(x_{A}\mid x_{B})$). So, $x_{A}$ can be constructed from $x_{B}$ (which is needed for the enumeration of $B_{1}^{\prime}$) and $p_{A}$, and therefore, $C(x_{A},x_{B})\leq C(p_{A},x_{B})$.

(c) $C(x_{B}\mid p_{A})=^{+}C(p_{A},x_{B})-C(p_{A})=^{+}C(x_{A},x_{B})-n_{A}$, by using the chain rule, and (a) and (b).

(d) $C(x_{B}\mid p_{A})=^{+}C(x_{A},x_{B})-n_{A}\leq^{+}(n_{A}+n_{B})-n_{A}=n_{B}$, by using (c) and the hypothesis. ∎

Now the assertions (1) and (2) follow, because by Fact 1, (d), $B_{2}$ is in the small regime, and by Fact 1 (c), the decoding procedure can enumerate $B_{1}$ since it knows $C(x_{A},x_{B})$ and $n_{A}$.

Finally, we move to the case of three senders.

3 Senders. This is the case stated in Theorem 3.3. We show how to reconstruct $x_{A},x_{B},x_{C}$ from $p_{A},p_{B},p_{C}$, if $n_{A},n_{B},n_{C}$ satisfy the inequalities (1). We can actually assume that $C(x_{A})>n_{A},C(x_{B})>n_{B},C(x_{C})>n_{C}$, because otherwise, if for example $C(x_{A})\leq n_{A}$, then $x_{A}$ can be reconstructed from $p_{A}$ as in the 1 Sender case, and we have reduced to the case of two senders. As in the 2 Senders case, it can be shown that $C(p_{A})=^{+}n_{A},C(p_{B})=^{+}n_{B},C(p_{C})=^{+}n_{C}$.

There are two cases to analyze.

Case 1. $C(x_{B}\mid p_{A})\leq n_{B}$ or $C(x_{C}\mid p_{A})\leq n_{C}$. Suppose the first relation holds. Then $x_{B}$ can be reconstructed from $p_{A},p_{B}$, by taking the small regime set for $G_{B}$,

[TABLE]

for which $x_{B}$ is a rich owner, and, therefore, with high probability owns $p_{B}$. In this way, we reduce to the 2 Senders case.

Case 2. $C(x_{B}\mid p_{A})>n_{B}$ and $C(x_{C}\mid p_{A})>n_{C}$. We show the following fact.

Fact 2**.**

1. (a)

$C(x_{B}\mid x_{C},p_{A})\leq^{+}n_{B}$* and $C(x_{C}\mid x_{B},p_{A})\leq^{+}n_{C}$ ,* 2. (b)

$C(x_{B},x_{C}\mid p_{A})\leq^{+}n_{B}+n_{C}$.

Proof.

(a) First note that $C(x_{C},p_{A})=^{+}C(p_{A})+C(x_{C}\mid p_{A})\geq^{+}n_{A}+n_{C}$. Then

[TABLE]

The other relation is shown in the obvious similar way.

(b)

[TABLE]

∎

Fact 2 shows that, given $p_{A}$, the complexity profile of $x_{B},x_{C}$ satisfies the requirements for the 2 Senders case, and therefore these two strings can be reconstructed from $p_{A},p_{B},p_{C}$. Next, since $n_{A}\geq C(x_{A}\mid x_{B},x_{C})$, $x_{A}$ can be reconstructed from $p_{A},x_{B},x_{C}$, as in the 1 Sender case.

Thus, both in Case 1 (which actually consists of two subcases Case 1.1 and Case 1.2) and in Case 2, $x_{A},x_{B},x_{C}$ can be reconstructed. There is still a problem: the decoding procedure needs to know which of the cases actually holds true. In the reference [Zim17], it is shown how to determine which case holds true, but here we present a solution that avoids this.

The decoding procedure launches parallel subroutines according to all three possible cases. The subroutine that works in the scenario that is correct produces $(x_{A},x_{B},x_{C})$. The subroutines that work with incorrect scenarios may produce other strings, or may not even halt. How can this troublesome situation be solved? Answer: by hashing. The senders, in addition to $p_{A},p_{B},p_{C}$, use a hash function $h$, and send $h(x_{A}),h(x_{B}),h(x_{C})$. The decoding procedure, whenever one of the parallel subroutines outputs a $3$-tuple, checks if the hash values of the tuple match $(h(x_{A}),h(x_{B}),h(x_{C}))$, and stops and prints that output the first time there is a match. In this way, the decoding procedure will produce with high probability $(x_{A},x_{B},x_{C})$.

Hashing. For completeness, we present one way of doing the hashing. By the Chinese Remainder Theorem, if $u_{1}$ and $u_{2}$ are $n$-bit numbers (in binary notation), then $u_{1}\bmod p=u_{2}\bmod p$ for at most $n$ prime numbers $p$. Suppose there are $s$ numbers $u_{1},\ldots,u_{s}$, having length $n$ in binary notation and we want to distinguish the hash value of $u_{1}$ from the hash values of $u_{2},\ldots,u_{s}$ with probability $1-\epsilon$. Let $t=(1/\epsilon)sn$ and consider the first $t$ prime numbers $p_{1},\ldots,p_{t}$. Pick $i$ randomly in $\{1,\ldots,t\}$ and define $h(u)=(p_{i},u\bmod p_{i})$. This isolates $u_{1}$ from $u_{2},\ldots,u_{s}$, in the sense that, with probability $1-\epsilon$, $h(u_{1})$ is different from any of $h(u_{2}),\ldots,h(u_{s})$. Note that the length of $h(u)$ is $O(\log n+\log s+\log(1/\epsilon))$. In our application above, $s=3$, corresponding to the three parallel subroutines, and $\epsilon$ can be taken to be $1/n^{2}$, and thus the overhead introduced by hashing is only $O(\log n)$ bits.

Removing the assumption that the decoding procedure knows the inputs’ complexity profile. So far, we have assumed that the decoding procedure knows the complexity profile of $x_{A},x_{B},x_{C}$. This assumption is lifted using a hash function $h$, akin to what we did above to handle the various cases for 3 Senders. The complexity profile $(C(x_{V})\mid V\subseteq\{A,B,C\},V\not=\emptyset\}$ is a $7$-tuple, with all components bounded by $O(n)$. The decoding procedures launches $O(n^{7})$ subroutines performing the decoding operation with known complexity profile, one for each possible value of the complexity profile. The subroutine using the correct value of the complexity profile will output $(x_{A},x_{B},x_{C})$ with high probability, while the other ones may produce different $3$-tuples, or may not even halt. Using the hash values $h(x_{A}),h(x_{B}),h(x_{C})$ (transmitted by senders together with $p_{A},p_{B},p_{C}$), the decoder can identify the correct subroutine in the same way as presented above. The overhead introduced by hashing is $O(\log n)$.

5 Constructing graphs with the rich owner property

We sketch the construction needed for the computable graph in Theorem 4.3. Recall that we use bipartite graphs of the form $G=(L=\{0,1\}^{n},R=\{0,1\}^{m},E\subseteq L\times R)$, in which every left node has degree $D=2^{d}$, and for every $x\in L$, the edges outgoing from $x$ are labeled with strings from $\{0,1\}^{d}$. The construction relies on randomness extractors, which have been studied extensively in computational complexity and the theory of pseudorandom objects. A graph of the above type is said to be a $(k,\epsilon)$ extractor if for every $B\subseteq L$ of size $|B|\geq 2^{k}$ and for every $A\subseteq R$,

[TABLE]

where $|E(B,A)|$ is the number of edges between vertices in $B$ and vertices in $A$. In a $(k,\epsilon)$ extractor, any subset $B$ of left nodes of size at least $2^{k}$, “hits” any set $A$ of right nodes like a random function: The fraction of edges that leave $B$ and land in $A$ is close to the density of $A$ among the set of right nodes. It is not hard to show that this implies the rich owner property in the large regime. To handle the small regime, we need graphs that maintain the extractor property when we consider prefixes of right nodes. Given a bipartite graph $G$ as above and $m^{\prime}\leq m$, the $m^{\prime}$-prefix graph $G^{\prime}$ is obtained from $G$ by merging right nodes that have the same prefix of length $m^{\prime}$. More formally, $G^{\prime}=(L=\{0,1\}^{n},R^{\prime}=\{0,1\}^{m^{\prime}},E^{\prime}\subseteq L\times R^{\prime})$ and $(x,z^{\prime})\in E^{\prime}$ if and only if $(x,z)\in E$ for some extension $z$ of $z^{\prime}$. Recall that we allow multiple edges between two nodes, and therefore the merging operation does not decrease the degree of left nodes.

Lemma 5.1**.**

For every $k\leq n$, and every $\epsilon>0$, there exists a constant $c$ and a computable graph $G=(L=\{0,1\}^{n},R=\{0,1\}^{k},E\subseteq L\times R)$ with left degree $D=cn/\epsilon^{2}$ such that for every $k^{\prime}\leq k$, the $k^{\prime}$-prefix graph $G^{\prime}=(L=\{0,1\}^{n},R^{\prime}=\{0,1\}^{k^{\prime}},E^{\prime}\subseteq L\times R^{\prime})$ is a $(k^{\prime},\epsilon)$ extractor.

Proof.

We show the existence of a graph with the claimed properties using the probabilistic method. Once we know that the graph exists, it can be constructed by exhaustive search. For some constant $c$ that will be fixed later, we consider a random function $f:\{0,1\}^{n}\times\{0,1\}^{d}\rightarrow\{0,1\}^{k}$. This defines the bipartite graph $G=(L=\{0,1\}^{n},R=\{0,1\}^{k},E\subseteq L\times R)$ in the following way: $(x,z)$ is an edge labeled by $y$ if $f(x,y)=z$. For the analysis, let us fix $k^{\prime}\in\{1,\ldots,k\}$ and let us consider the graph $G^{\prime}=(L,R^{\prime},E^{\prime}\subseteq L\times R^{\prime})$ that is the $k^{\prime}$-prefix of $G$. Let $K^{\prime}=2^{k^{\prime}}$ and $N=2^{n}$. Let us consider $B\subseteq\{0,1\}^{n}$ of size $|B|\geq K^{\prime}$, and $A\subseteq R^{\prime}$. For a fixed $x\in B$ and $y\in\{0,1\}^{d}$, the probability that the $y$-labeled edge outgoing from $x$ lands in $A$ is $|A|/|R^{\prime}|$. By the Chernoff bounds,

[TABLE]

The probability that relation (2) fails for some $B\subseteq\{0,1\}^{k^{\prime}}$ of size $|B|\geq K^{\prime}$ and some $A\subseteq R^{\prime}$ is bounded by $2^{K^{\prime}}\cdot{N\choose K^{\prime}}\cdot 2^{-\Omega(K^{\prime}\cdot D\cdot\epsilon^{2})}$, because $A$ can be chosen in $2^{K^{\prime}}$ ways, and we can consider that $B$ has size exactly $K^{\prime}$ and there are ${N\choose K^{\prime}}$ possible choices of such $B$’s. If $D=cn/\epsilon^{2}$ and $c$ is sufficiently large, the above probability is less than $(1/4)2^{-k^{\prime}}$. Therefore the probability that relation (2) fails for some $k^{\prime}$, some $B$ and some $A$ is less than $1/4$. It follows, that there exists a graph that satisfies the hypothesis. ∎

Let $G=(L,R,E\subseteq L\times R)$ be the $(k,\epsilon)$-extractor from Lemma 5.1. Let $\delta=(2\epsilon)^{1/2}$. As hinted in our discussion above, by manipulating relation (2), we can show that for every $B\subseteq L$ of size $|B|>2^{k}$, $(1-\delta)$ fraction of nodes in $B$ are rich owners with respect to $B$. This proves the rich owner property for sets in the large regime. If $B$ is in the small regime, then $B$ has size $2^{k^{\prime}}$ for some $k^{\prime}<k$ (for simplicity, we assume that the size of $B$ is a power of two). Let us consider $G^{\prime}$, the $k^{\prime}$-prefix of $G$. As above, in $G^{\prime}$, $(1-\delta)$ fraction of elements $x$ in $B$ are rich owners with respect to $B$. Recall that this means that if $x$ is a rich owner then $(1-\delta)$ fraction of its neighbors have $B$-degree bounded by $s$, where $s=(2/\delta^{2})|B|\cdot D/|R^{\prime}|=O(n/\epsilon^{3})$. Using the same hashing technique, we can “split” each edge into $\rm{poly}(n/\epsilon)$ new edges. More precisely, an edge $(x,z)$ in $G^{\prime}$ is transformed into $\ell=(1/\delta)sn$ new edges, $(p_{1},x\bmod p_{1},z),\ldots,(p_{\ell},x\bmod p_{\ell},z)$, where, as above, $p_{i}$ is the $i$-th prime number. If $x$ is a rich owner then $(1-2\delta)$ of its “new” neighbors (obtained after splitting) have $B$-degree equal to one, as desired, because hashing isolates $x$ from the other neighbors of $z$. The $B$-degree of these right nodes continues to be one also in $G$, because when merging nodes to obtain $G^{\prime}$ from $G$, the right degrees can only increase. Note that the right nodes in $G$ have as labels the $k$-bit strings, and after splitting we need to add to the labels the hash values $(p_{i},x\bmod p_{i})$, which are of length $O(\log n/\epsilon)$, and this is the cause for the $O(\log n/\delta)$ overhead in Theorem  4.3.

The explicit graph in Theorem 4.3 is obtained in the same way, except that instead of the “prefix” extractor from Lemma 5.1 we use the Raz-Reingold-Vadhan extractor [RRV99].

6 Note

This paper is dedicated to the memory of Professor Solomon Marcus. In the 1978 freshman Real Analysis class at the University of Bucharest, he asked several questions (on functions having pathological properties regarding finite variation). This has been my first contact with him, and, not coincidentally, also the first time I became engaged in a type of activity that resembled mathematical research. Over the years, we had several discussions, on scientific but also on rather mundane issues, and each time, without exception, I was stunned by his encyclopedic knowledge on diverse topics, including of course various fields of mathematics, but also literature, social sciences, philosophy, and whatnot.

The utilitarian function of mathematics is in most cases a consequence of its cognitive function, but the temporal distance between the cognitive moment and the utilitarian one is usually imprevisible.

Solomon Marcus