Guessing Individual Sequences: Generating Randomized Guesses Using Finite-State Machines

TL;DR

This paper studies a finite-state machine approach to guessing individual sequences, linking the guessing difficulty to sequence compressibility and demonstrating the effectiveness of a modified Lempel-Ziv algorithm in this context.

Contribution

It introduces the finite-state guessing exponent and connects it to sequence compressibility, showing that a modified Lempel-Ziv algorithm asymptotically achieves this bound.

Findings

Finite-state guessing exponent relates to sequence compressibility.

Modified Lempel-Ziv algorithm asymptotically achieves the guessing exponent.

Results extend to scenarios with side information sequences.

Abstract

Motivated by earlier results on universal randomized guessing, we consider an individual-sequence approach to the guessing problem: in this setting, the goal is to guess a secret, individual (deterministic) vector $x^n=(x_1,\ldots,x_n)$, by using a finite-state machine that sequentially generates randomized guesses from a stream of purely random bits. We define the finite-state guessing exponent as the asymptotic normalized logarithm of the minimum achievable moment of the number of randomized guesses, generated by any finite-state machine, until $x^n$ is guessed successfully. We show that the finite-state guessing exponent of any sequence is intimately related to its finite-state compressibility (due to Lempel and Ziv), and it is asymptotically achieved by the decoder of (a certain modified version of) the 1978 Lempel-Ziv data compression algorithm (a.k.a. the LZ78 algorithm), fed by purely random bits. The results are also extended to the case where the guessing machine has access to a side information sequence, $y^n=(y_1,\ldots,y_n)$, which is also an individual sequence.

Equations

$$t_i$$

$$v_i$$

$${\hat{x}}_i$$

$$z_{i+1}$$

$$\gamma_{s}(x^{n})=\min_{Q\in{\cal Q}(s)}\frac{\ln\mbox{\boldmath$E$}\{[G_{Q}(x^{n})]^{\zeta}\}}{n},$$

$$\gamma_{s}(\mbox{\boldmath$x$})=\limsup_{n\to\infty}\gamma_{s}(x^{n}),$$

$$\gamma(\mbox{\boldmath$x$})=\lim_{s\to\infty}\gamma_{s}(\mbox{\boldmath$x$}).$$

$$0\to \text{mbox}ab10\to \text{mbox}bac11\to \text{mbox}ca.$$

$$b_i=p({\sigma }_i,x_i),{\sigma }_{i+1}=q({\sigma }_i,x_i),i=1,2,...$$

$${\rho }_E(x^n)=\frac{L_E(b^n)}{n}.$$

$${\rho }_K(x^n)=\underset{E\in \mathcal{E}(K)}{\min }{\rho }_E(x^n),$$

$$\rho_{K}(\mbox{\boldmath$x$})=\limsup_{n\to\infty}\rho_{K}(x^{n}),$$

$$\rho(\mbox{\boldmath$x$})=\lim_{K\to\infty}\rho_{K}(\mbox{\boldmath$x$}).$$

$${\rho }_K(x^n)\ge \frac{[c(x^n)+K^2]}{n}\log \left[\frac{c(x^n)+K^2}{4K^2}\right]+\frac{2K^2}{n},$$

$${\gamma }_s(x^n)$$

$$\gamma(\mbox{\boldmath$x$})=\zeta\cdot\rho(\mbox{\boldmath$x$}).$$

$$P({\hat{x}}^n|z_1)=\sum _{z_2,z_3,\dots ,z_{n+1}}\prod _{i=1}^{n}P({\hat{x}}_i,z_{i+1}|z_i),$$

$$\mbox{\boldmath$E$}\{[G_{Q}(x^{n})]^{\zeta}\}=\sum_{k=1}^{\infty}k^{\zeta}[1-P(x^{n})]^{k-1}P(x^{n}).$$

$$\sum _{k=1}^{\infty }{k}^{\zeta }(1-q{)}^{k-1}\cdot q$$

$$\displaystyle\mbox{\boldmath$E$}\{[G_{Q}(x^{n})]^{\zeta}\}$$

$$P({\hat{x}}^n,z^m)=\prod _{i=0}^{m-1}P({\hat{x}}_{i\ell +1}^{i\ell +\ell },z_{i\ell +\ell +1}|z_{il+1}).$$

$$\log |\mathcal{T}({\hat{x}}^n|z^m)|\ge m[{\hat{H}}_{\ell }(x^n)-\log (s^{2}e)],$$

$${\hat{H}}_{\ell }({\hat{x}}^n)=-\sum _{a^{\ell }\in {\mathcal{X}}^{\ell }}\hat{P}(a^{\ell })\log \hat{P}(a^{\ell }),$$

$$\hat{P}(a^{\ell })=\frac{1}{m}\sum _{i=0}^{m-1}\mathcal{I}\{{\hat{x}}_{i\ell +1}^{i\ell +\ell }=a^{\ell }\},a^{\ell }\in {\mathcal{X}}^{\ell }$$

$$1\ge \sum _{{\tilde{x}}^n\in \mathcal{T}({\hat{x}}^n|z^m)}P({\tilde{x}}^n,z^m)=|\mathcal{T}({\hat{x}}^n|z^m)|\cdot P({\hat{x}}^n,z^m),$$

$$P({\hat{x}}^n)$$

$$-\log P({\hat{x}}^n)\ge m[{\hat{H}}_{\ell }(x^n)-\log (s^{3}e)].$$

$$n{\rho }_{K(\ell )}({\hat{x}}^n)$$

$$n{\rho }_{K(\ell )}({\hat{x}}^n)\le m{\hat{H}}_{\ell }({\hat{x}}^n)+m.$$

$$-\log P({\hat{x}}^n)\ge n{\rho }_{K(\ell )}({\hat{x}}^n)-m\log (2s^{3}e),$$

$$n{\rho }_{K(\ell )}({\hat{x}}^n)$$

Full text

Index Terms: guessing exponent, individual sequences, sequence complexity, finite–state machine, Lempel–Ziv algorithm, incremental parsing, side information, randomized guessing.

The Andrew & Erna Viterbi Faculty of Electrical Engineering

Technion - Israel Institute of Technology

Technion City, Haifa 32000, ISRAEL

E–mail: merhav@ee.technion.ac.il

1 Introduction

Consider the problem of guessing the realization of a finite–alphabet random vector $X^{n}=(X_{1},\ldots,X_{n})$ using a series of yes/no questions of the form: “Is $X^{n}=x^{n}(1)$?”, “Is $X^{n}=x^{n}(2)$?”, and so on, until a positive response is received. Given a distribution on $X^{n}$, a commonly used performance metric for the guessing problem is the expected number of guessing trials required until $X^{n}$ is guessed correctly, or more generally, a general moment of this number.

The design of guessing strategies with the quest of minimizing the moments of the number of guesses has several applications in information theory and related fields. One of them, for example, is sequential decoding, as shown by Arikan [1], who built on the pioneering earlier work of Massey [6] and related the best achievable guessing moment to the Rényi entropy. More modern applications of the guessing problem evolve around aspects of information security, in particular, guessing passwords or decrypting messages protected by random keys. For example, one may submit a sequence of guessing queries in attempt to crack passwords – see, e.g., [10, Introduction] (as well as [11] and other references therein) for a brief, yet quite comprehensive review on guessing and security, as well as for some general historical overview of prior work on the guessing problem in its large plethora of variants and extensions.

One of the main results in [11] is about devising optimal randomized guessing strategies, where instead of designing a particular, deterministic guessing list in advance, one randomly draws independent guesses according to a carefully chosen probability distribution of $n$–vectors, with the motivation that such a randomized strategy saves the need of storing in memory long guessing lists and it also saves the need for synchronization between the guesses of two or more agents who attempt to crack the same password from different IP addresses in parallel. It turns out that with a clever choice of the randomized guessing distribution, the resulting moment of the number of guesses (w.r.t. the randomness of both $X^{n}$ and the guesses themselves) is exponentially the same as that of the optimal deterministic guessing strategy. In a later work [10], this finding was further strengthened in two different ways: first, the framework was extended from that of a discrete memoryless source (that governs $X^{n}$) to the much more general non–unifilar, finite–state source (hidden Markov source model). Secondly, a universal randomized guessing distribution was proposed, which is independent of the unknown parameters of this finite–state source, as well as the moment order of the number of guesses. The universal random guessing distribution, $P(x^{n})$, proposed in [10], was proportional to $2^{-LZ(x^{n})}$, where $LZ(x^{n})$ designates the length (in bits) of the compressed version of $x^{n}$ according to the LZ78 data compression algorithm [14]. Moreover, practical algorithms for efficiently implementing this distribution were proposed in [10]. Finally, these results were also extended to account for the availability of side information, $Y^{n}$, that is correlated to $X^{n}$, under a probabilistic model where the sequence of pairs $\{(X_{i},Y_{i})\}$ emerges from a non–unifilar finite–state source.

Motivated by those results of [10], in this work, we make an additional step towards generality, by completely dropping the probabilistic assumption concerning the $n$–vector to be guessed. In particular, we assume that it is an individual (deterministic), finite–alphabet vector, which will be denoted by $x^{n}$. We also assume that the independent random guesses are generated sequentially, using a finite–state machine, fed by a sequence of purely random bits.111The supply of random bits is assumed unlimited. Inspired by the individual–sequence approach to data compression, pioneered by Ziv and Lempel, [14], and followed later in the context of other tasks, like gambling [3], prediction [4], and encryption [8], we define the finite–state guessing exponent as the minimum asymptotic exponential rate of the expectation of a given power of the number of guesses, that is achievable by any finite–state machine, and propose a universal randomized guessing scheme that asymptotically achieves the finite–state guessing exponent. Similarly as in [3] and [8] (but in contrast to [4]), we show that the finite–state guessing exponent is very intimately related to the finite–state compressibility of the sequence to be guessed.

The proposed achievability scheme is basically the same as in [10], which was based on the simple idea of feeding the LZ78 decoder by purely random bits. While such a scheme cannot be realized using a finite–state machine, a simple twist can be offered, similary as done in [14] in the context of compression: by employing this randomized guessing scheme repeatedly and resetting its memory at the beginning of every new block (however long), it becomes implementable as a finite–state machine, and it still achieves the finite–state guessing exponent in the limit of an increasing number of states. We therefore prove that the same achievability scheme as in [10] is asymptotically optimal, not only in the probabilistic setting, but also in the individual–sequence setting that we define here. Finally, we outline how these findings extend (with a few twists) to the case where the guessing machine has access to a (deterministic) side information sequence.

The outline of the paper is as follows. In Section 2, we formalize the problem setting and spell out the objectives. In Section 3, we assert the converse theorem and prove it. In Section 4, we present the achievability scheme and prove the direct theorem. Finally, in Section 5, we outline the main modifications needed in order to extend the model and the results to the case where side information is available to the guessing machine.

2 Problem Setting

A finite–state guessing machine (FSGM) is defined by a sixtuplet $Q=({\cal U},{\cal X},{\cal Z},\Delta,f,g)$, where ${\cal U}=\{0,1\}$ is the binary input alphabet, ${\cal X}$ is a finite output alphabet of size $\alpha$, ${\cal Z}$ is a finite set of states, $\Delta:{\cal Z}\to\{0,1,2,\ldots\}$ defines the number of input bits processed at each state, $f:{\cal Z}\times{\cal U}^{*}\to{\cal X}$ is the output function, and $g:{\cal Z}\times{\cal U}^{*}\to{\cal Z}$ is the next–state function, where ${\cal U}^{*}$ is a set of variable–length binary strings. When a binary sequence, \mbox{\boldmathu}=u_{1},u_{2},\ldots, $u_{i}\in{\cal U}$, $i=1,2,\ldots$, drawn from the binary symmetric source (BSS), is fed into an FSGM $Q$, it produces an output sequence \hat{\mbox{\boldmathx}}^{n}=(\hat{x}_{1},\hat{x}_{2},\ldots,\hat{x}_{n})\in{\cal X}^{n}, while passing through a sequence of states, $z_{1},z_{2},\ldots,z_{n}$, according to the following recursive equations, implemented for $i=1,2,\ldots,n$,

[TABLE]

where, without loss of generality, $z_{1}$ is assumed to be some fixed member of ${\cal Z}$. An FSGM $Q$ with $s$ states, or an $s$–state guessing machine, is one with $|{\cal Z}|=s$.

In the guessing game between Alice and Bob, Alice has access to a certain secret $n$–vector $x^{n}=(x_{1},\ldots,x_{n})\in{\cal X}^{n}$, while Bob is unaware of $x^{n}$, but is equipped with an FSGM $Q$. In each guessing round, Bob activates $Q$ by feeding it with a sequence of purely random bits, $u_{1},u_{2},\ldots$, until an output sequence of length $n$, $\hat{x}^{n}=(\hat{x}_{1},\hat{x}_{2},\ldots,\hat{x}_{n})$, is obtained, which is then submitted to Alice as a guess. Alice in her turn compares $\hat{x}^{n}$ to $x^{n}$ and returns an affirmative response if they match, and a negative one if they do not. In the former case, $x^{n}$ has been guessed successfully and the guessing process terminates. In the latter case, a new guessing round takes place, using new, independent random bits, and so on, until $x^{n}$ is guessed successfully.

Let $G_{Q}(x^{n})$ be the random number of guessing rounds needed for $Q$ until success. For a given $\zeta>0$, define

[TABLE]

where ${\cal Q}(s)$ is the set of all FSGMs with no more than $s$ states. In order to define asymptotics, consider an infinite sequence \mbox{\boldmathx}=(x_{1},x_{2},\ldots), whose components take on values in ${\cal X}$. We define the asymptotic $s$–state guessing exponent as

[TABLE]

and finally, the finite–state guessing exponent of $x$ is defined as

[TABLE]

While the minimizing FSGM $Q^{*}$ of eq. (5) depends, in general, on $x^{n}$, our objective is to devise a universal, sequential, randomized guessing scheme that is independent of $x^{n}$, and yet it asymptotically achieves $\gamma_{s}(x^{n})$ in the limit of large $n$, followed by the limit of large $s$, and therefore it achieves also \gamma(\mbox{\boldmathx}). Moreover, this universal guessing scheme will not depend on the moment order $\zeta$ either.

Two observations regarding the above defined model of the FSGM are in order.

1.  As can be seen in eqs. (1)–(4), at each cycle $i$, the FSGM processes $\Delta(z_{i})$ new input bits, in other words, a number of bits that depends solely on the current state, $z_{i}$. We could have defined the model to be seemingly more general, where the number of input bits depends, not only on $z_{i}$, but also on a certain number of the next incoming input bits, $u_{t_{i-1}+1},u_{t_{i-1}+2},\ldots$, in the following manner. Consider a situation where for each state $z\in{\cal Z}$, one defines a binary tree, $T(z)$, and define $\Delta(z_{i},u_{t_{i-1}+1},u_{t_{i-1}+2},\ldots)$ to be the number of branches of the path from the root of $T(z)$ down to the leaf pertaining to the trail associated with $u_{t_{i-1}+1},u_{t_{i-1}+2},\ldots$. On the other hand, if one does not care about “wasting” input bits, this model can be formalized in the above defined framework by re–defining $\Delta(z)$ to be the length of the path from the root to the deepest leaf and then extending $T(z)$ to be a full binary tree of depth $\Delta(z)$, where $f$ and $g$ are defined to be the same for all descendants of every given leaf of the original tree, $T(z)$. For example, if $T(z)$ originally contains the three leaves, corresponding to the binary paths ‘0’, ‘10’ and ‘11’, then we extend the path ‘0’ to its two children, ‘00’ and ‘01’ to obtain a full binary tree of four leaves and depth $\Delta(z)=2$, but we let $f(z,00)=f(z,01)$ and $g(z,00)=g(z,01)$ (see also the example in comment no. 2 below), so that the second bit after ‘0’ is not really used and hence is immaterial. Therefore, the only difference between the FSGM with the extended tree and the original FSGM is that the second bit of ‘00’ and ‘01’ is “wasted”, but since we are assuming, in this paper, that resources of randomness are unlimited, this is inconsequential. The reason we prefer the model of $\Delta(z_{i})$ over the model of $\Delta(z_{i},u_{t_{i-1}+1},u_{t_{i-1}+2},\ldots)$ is just its relative simplicity.

2. The above defined model of the FSGM is general enough to operate as any (state–dependent) mapping from variable–length binary input strings to variable–length strings of symbols, such as decoders corresponding to variable–to–variable length encoders for data compression. This is the case when we allow $\Delta(z)=0$ at some states, as it enables the finite–state machine to idle between successive readings of chunks of input bits. To implement such a variable–to–variable length mapping in the framework of our model, the system works as follows. Upon receiving a variable–length binary input string $v_{i}$, the system passes along a certain sequence of states, where at each state it outputs one output symbol without reading any new input bits ($\Delta(z_{j})=0$), until the entire output word is produced. As a simple example, consider the variable–to–variable length mapping

[TABLE]

The state transition diagram of the associated FSM is depicted in Fig. 1, as a graph whose vertices represent the states and whose edges designate the state transitions.

Each state transition is defined by a pair $(z,v)$, $z\in{\cal Z}$, $v\in{\cal U}^{\Delta(z)}$, and its edge is labeled in Fig. 1 by the corresponding output $\hat{x}=f(z,v)$, followed by a slash, and followed in turn by the contents of $v$. For instance, states and A and B represent the assignment $0\to\mbox{ab}$ or equivalently, the pair of assignments $00\to\mbox{ab}$ and $01\to\mbox{ab}$ (see comment no. 1 above). The transition from state A to state B is labeled by “a/null”, which means $\Delta(\mbox{A})=0$ and $v$ is null (no new bits are read by the system), and the output is $\mbox{a}=f(\mbox{A},\mbox{null})$. This means that after visiting at state A, the machine must pass to state B (as $\mbox{B}=g(\mbox{A},\mbox{null})$), without processing input bits. From state B, all outgoing edges are associated with the output “b”, and $\Delta(\mbox{B})=2$ new bits are read for the next round of the mapping (8): if the first bit is ‘0’, then we are back to state A, otherwise, we pass to state C or state F, to map ‘10’ or ‘11’, respectively, and we proceed similarly as before.

3 Converse Theorem

3.1 Preliminaries

Before presenting the converse theorem and its proof, we need to recall the notions of finite–state compressibility and string parsing from [14] (which inspired the above definition of the FSGM). To make the presentation self–contained, we provide the definitions of these terms here, but with a slightly different notation, not to confuse with the above notation of the FSGM.

A $K$–state encoder $E$ is defined by a quintuple $({\cal X},{\cal B},\Sigma,f,g)$, where ${\cal X}$ is the alphabet of size $\alpha$ of the source sequence to be compressed, ${\cal B}$ is a finite set of binary words (possibly of different lengths, including the null word for idling), $\Sigma$ is a finite set of $K$ states, $p:\Sigma\times{\cal X}\to{\cal B}$ is the encoder output function, and $q:\Sigma\times{\cal X}\to\Sigma$ is the next–state function. When an input sequence $(x_{1},x_{2},...)$ is fed sequentially into $E$, the encoder outputs a sequence of binary words $(b_{1},b_{2},\ldots)$, $b_{i}\in{\cal B}$, while going through a sequence of states $(\sigma_{1},\sigma_{2},\ldots)$, $\sigma_{i}\in\Sigma$, according to

[TABLE]

where $\sigma_{i}$ is the state of $E$ at time instant $i$ and where the initial state, $\sigma_{1}$, is assumed a fixed member of $\Sigma$. The decoder receives the compressed bit–stream, $b_{1},b_{2},\ldots$, and reconstructs $x_{1},x_{2},\ldots$. A finite–state encoder $E$ is said to be information lossless (IL) if for all $\sigma_{1}\in\Sigma$ and all $x^{n}\in{\cal X}^{n}$, $n\geq 1$, the triple $(\sigma_{1},\sigma_{n+1},b^{n})$ (with $b^{n}\stackrel{{\scriptstyle\Delta}}{{=}}(b_{1},\ldots,b_{n})$) uniquely determines $x^{n}$, where $\sigma_{n+1}$ and $b^{n}$ are obtained by iterating eq. (9) with $\sigma_{1}$ and $x^{n}$ as inputs. The length function associated with $E$ is defined as $L_{E}(b^{n})=\sum_{i=1}^{n}l(b_{i})$, where $l(b_{i})$ is the length of the binary string $b_{i}\in{\cal B}$. The compression ratio of $x^{n}$ w.r.t. $E$ is defined as222Note that here, unlike in [14], we define the compression ratio without normalization by $\log\alpha$, where $\alpha$ is the source alphabet size.

[TABLE]

Next, we define

[TABLE]

where ${\cal E}(K)$ is the class of all IL encoders with no more than $K$ states. Furthermore, for an infinite sequence \mbox{\boldmathx}=(x_{1},x_{2},\ldots), let

[TABLE]

and finally,

[TABLE]

The following converse theorem was asserted and proved in [14] in the context of data compression.

Theorem 1

[14, Theorem 1]** For every $x^{n}\in{\cal X}^{n}$,

[TABLE]

where $c(x^{n})$ is the largest number of distinct strings (or phrases) whose concatenation forms $x^{n}$.

3.2 Main Result

Returning to our guessing problem, our converse theorem relates the finite–state guessing exponent of $x^{n}$ to its finite–state compressibility.

Theorem 2

Let $Q$ be an arbitrary FSGM with $s$ states and let $x^{n}$ be a sequence generated from $Q$ with probability less than $1/2$. Then,

[TABLE]

where $K(\ell)\stackrel{{\scriptstyle\Delta}}{{=}}(\alpha^{\ell+1}-1)/(\alpha-1)$ and $\delta_{n}(s)$ tends to zero uniformly as $n\to\infty$ for any fixed $s$.

Discussion. The condition that $x^{n}$ is generated from $Q$ with probability less than $1/2$ is purely technical and it is almost trivially always met. It means that at least one input bit is utilized for the generation of $x^{n}$. Theorem 2 provides two different lower bounds on $\gamma_{s}(x^{n})$. The first one relates the $s$–state guessing exponent with the $K$–state compressibility of $x^{n}$, and it implies also an asymptotic version of our converse theorem, which is \gamma(\mbox{\boldmathx})\geq\zeta\cdot\rho(\mbox{\boldmathx}). This lower bound will be shown to be achieved asymptotically by a certain sequence of FSGMs whose number of states goes to infinity after $n\to\infty$, thus establishing also the reversed inequality \gamma(\mbox{\boldmathx})\leq\zeta\cdot\rho(\mbox{\boldmathx}), and hence an equality,

[TABLE]

The second lower bound, which is in fact a corollary of the first one, and hence is weaker, has the advantage that it is computable. It can be achieved by a randomized guessing machine whose number of states is unlimited. The details are deferred to the next section, where the achievability results will be asserted and proved.

Proof of Theorem 2. Let $Q$ be an arbitrary FSGM with $s$ states. First, observe that since the input bits are i.i.d., the output of $Q$ is a non–unifilar333A non–unifilar finite–state source is a finite–state source where the underlying state sequence is hidden, namely, it cannot be recovered from the source sequence alone. A unifilar source is obtained as a special case, where eq. (4) is replaced by the equation $z_{i+1}=g(z_{i},\hat{x}_{i})$. Note that since $\hat{x}_{i}=f(z_{i},v_{i})$, then eventually, here too $z_{i+1}$ is a function of $(z_{i},v_{i})$, but not every function of $(z_{i},v_{i})$ can be represented as a composition of $f$ and $g$ in the form, $z_{i+1}=g(z_{i},f(z_{i},v_{i}))$. Thus, our model is more general than the model of unifilar finite–state sources. finite–state source, a.k.a. a hidden Markov source. In particular,

[TABLE]

where $P(\hat{x},z^{\prime}|z)=m(\hat{x},z^{\prime}|z)\cdot 2^{-\Delta(z)}$, $m(\hat{x},z^{\prime}|z)$ being the number of binary strings $\{v\}$ of length $\Delta(z)$ such that $f(z,v)=\hat{x}$ and $g(z,v)=z^{\prime}$, as each such binary string has probability $2^{-\Delta(z)}$. Since $z_{1}$ is assumed fixed, the conditioning on $z_{1}$ will henceforth be dropped. Now, since the guesses are statistically mutually independent,

[TABLE]

Consider now the following chain of inequalities, holding for any $q\in(0,1]$:

[TABLE]

where in the last step, we have used the inequality $\ln(1+t)\geq\frac{t}{1+t}$, which holds for all $t>-1$. Thus, with the assignment $q=P(x^{n})$, and since it is assumed that $P(x^{n})\leq 1/2$, we have

[TABLE]

Assume that $\ell$ divides $n$ and consider the partition of $\hat{x}^{n}$ into $m=n/\ell$ non–overlapping blocks of length $\ell$, $\hat{x}^{n}=(\hat{x}_{1}^{\ell},\hat{x}_{\ell+1}^{2\ell},\ldots,\hat{x}_{n-\ell+1}^{n})$, where here and throughout the sequel, the notation $\hat{x}_{i}^{j}$ ($i<j$) denotes the string segment $(\hat{x}_{i},\hat{x}_{i+1},\ldots,\hat{x}_{j})$ (and a similiar notation applies also to other vectors). We also define the diluted sequence of states $z^{m}\stackrel{{\scriptstyle\Delta}}{{=}}(z_{1},z_{\ell+1},z_{2\ell+1},\ldots,z_{n-\ell+1})$. Then, it follows from eq. (18) that

[TABLE]

Let ${\cal T}(\hat{x}^{n}|z^{m})$ denote the set of vectors in ${\cal X}^{n}$ that are obtained by permuting different $\ell$–blocks that begin at the same state, $z$, and end at the same state, $z^{\prime}$. Obviously, all members of ${\cal T}(\hat{x}^{n}|z^{m})$ have the same joint probability with $z^{m}$. A simple combinatorial argument (see, e.g., [7, eq. (A.11)]) yields that

[TABLE]

where $\hat{H}_{\ell}(x^{n})$ is the entropy associated with the empirical distribution of the non–overlapping $\ell$–blocks of $\hat{x}^{n}$, that is,

[TABLE]

where

[TABLE]

${\cal I}\{\hat{x}_{i\ell+1}^{i\ell+\ell}=a^{\ell}\}$ being the indicator function for the event $\{\hat{x}_{i\ell+1}^{i\ell+\ell}=a^{\ell}\}$. Now, since

[TABLE]

it follows that

[TABLE]

or, equivalently,

[TABLE]

It remains to lower bound the r.h.s. in terms of the finite–state compressibility. Consider a Shannon code w.r.t. an arbitrary probability distribution $F$ of $\ell$–vectors, that is, a code that assigns $\lceil-\log F(x^{\ell})\rceil$ bits to the lossless compression of $x^{\ell}$. Similarly as argued in [9, p. 2245, right column], such a code can be implemented by an IL finite–state encoder with $\sum_{j=0}^{\ell}\alpha^{j}=(\alpha^{\ell+1}-1)/(\alpha-1)=K(\ell)$ states, and so, by the definition of the $K$–state compressibility,

[TABLE]

and since this is true for every probability distribution $F$ on ${\cal X}^{\ell}$, the r.h.s. may be minimized w.r.t. $F$, to obtain

[TABLE]

Combining this with eq. (28), we obtain

[TABLE]

which, together with eq. (21), proves the first inequality of the converse theorem, since $\ell$ is an arbitrary divisor of $n$.

Finally, the second lower bound of the converse theorem is obtained from [14, Theorem 1], as follows.

[TABLE]

where $\epsilon_{n}\to 0$, and the last step follows from the inequality $c(x^{n})\leq n\log\alpha/[(1-\epsilon_{n})\log n]$ [5, Theorem 2], [2, Lemma 13.5.3, p. 450]. It follows that the second lower bound of Theorem 2 holds with

[TABLE]

This completes the proof of Theorem 2.

4 Direct Theorem

4.1 Preliminaries

Before presenting our direct theorem (achievability), we need to recall a few more terms and facts from [14].

The incremental parsing procedure of the LZ78 algorithm is a procedure of sequentially parsing a vector $x^{n}$ such that each new phrase is the shortest string that has not been encountered before as a parsed phrase, with the possible exception of the last phrase, which might be incomplete. For example, the incremental parsing of the vector $x^{15}=\mbox{abbabaabbaaabaa}$ is a,b,ba,baa,bb,aa,ab,aa. Let $c_{\mbox{\tiny LZ}}(x^{n})$ denote the number of phrases in $x^{n}$ resulting from the incremental parsing procedure. Obviously, $c_{\mbox{\tiny LZ}}(x^{n})\leq c(x^{n})+1$ [5, Theorem 2], [14, eq. (6)], as $c(x^{n})$ was defined above as the maximum number of distinct phrases. Let $LZ(x^{n})$ denote the length of the LZ78 binary compressed code for $x^{n}$. According to [14, Theorem 2],

[TABLE]

where $\epsilon(n)$ clearly tends to zero as $n\to\infty$, at the rate of $1/\log n$.

4.2 Main Result

Returning to the guessing problem, our direct theorem is as follows.

Theorem 3

• (a)

Consider the random guessing distribution

[TABLE]

Then,

[TABLE]

• (b)

Let $\ell$ divide $n$ and consider the product form random guessing distribution,

[TABLE]

Then, for every positive integer $K$,

[TABLE]

Discussion. Part (a) of Theorem 3 is an achievability result that is matching the second lower bound in Theorem 2. However, this is incompatible with the framework of finite–stage machines since this random guessing distribution cannot be implemented with a finite–state machine as $n$ grows without bound. The random guessing distribution of part(b), on the other hand, can be implemented using an FSGM with no more than $\ell\cdot\alpha^{\ell}$ states, and it is a matching achievability result to the first lower bound of Theorem 2. It should be pointed that even the random guessing distribution of part (a) can be implemented efficiently using practical algorithms, as described in [10]. The upper bound of part (b) is meaningful when $K^{2}\ll\ell$.

Theorems 2 and 3 together tell us that essentially the best achievable guessing moment \mbox{\boldmathE}\{[G(x^{n})]^{\eta}\} is of the exponential order of $2^{\zeta c(x^{n})\log c(x^{n})}$. In general, when the $\eta$-th moment of a random variable behaves like $A^{\eta}$ for some positive constant $A$ that is independent of $\eta$, and every $\eta>0$, it indicates that this random variable is (nearly) degenerate. In other words, it concentrates very rapidly around its mean. Indeed, it can easily be shown very similarly444In eq. (17) of [10], this it is shown for the empirical entropy (instead of $c(x^{n})\log c(x^{n})$, but the proof for $c(x^{n})\log c(x^{n})$ is exactly the same. as in [10, eq. (17)], that the probability of the event $\{G(x^{n})\geq 2^{c(x^{n})\log c(x^{n})+n\epsilon}\}$ decays double exponentially rapidly for the optimal guessing distribution, provided that $\epsilon>0$.

Proof of Theorem 3. Part (a) is almost a restatement of [10, Theorem 3]. The proof is therefore almost identical to the proof of that result. The only difference is that here, since $x^{n}$ is a given deterministic vector, the final step in [10, proof of Theorem 3], of taking the expectation w.r.t. the randomness of $x^{n}$, is now omitted, and the expectation of $[G_{Q}(x^{n})]^{\zeta}$ is taken only w.r.t. the randomness of the guesses, which as is shown in [10, Lemma 1], behaves like $[P(x^{n})]^{-\zeta}$, where $P(\cdot)$ is the random guessing distribution.

As for part(b) of Theorem 3, since $LZ(\cdot)$ is a uniquely decipherable code, it satisfies Kraft’s inequality, and so,

[TABLE]

where the last step follows from eq. (34) applied to $\ell$–vectors. It remains to show that $\sum_{i=0}^{n/\ell-1}c(\hat{x}_{\ell i+1}^{\ell i+\ell})\log c(\hat{x}_{\ell i+1}^{\ell i+\ell})$ is essentially no larger than $\rho_{K}(\hat{x}^{n})$ for some $K$ that can be chosen arbitrarily large, provided that $n\gg\ell$ and $\ell$ is large enough. Consider the following chain of inequalities for a given positive integer $K$:

[TABLE]

where the first inequality follows from [14, Theorem 1] applied to $\ell$–vectors. Thus,

[TABLE]

and so,

[TABLE]

This completes the proof of Theorem 3.

5 Side Information

We now consider the extended setting where a deterministic side information vector, $y^{n}$, is available to the randomized guessing machine. Since most of the ideas and techniques extend quite straightforwardly, we only outline the differences compared to the case without side information.

We now define the model as follows. An FSGM is defined by a set $Q=({\cal U},{\cal X},{\cal Y},{\cal Z},\ell,f,g,\Delta)$, where ${\cal U}$, ${\cal X}$, and ${\cal Z}$ are as before, ${\cal Y}$ is the finite alphabet of size $\beta$ associated with the side information, $\ell$ is a positive integer, $f:{\cal Z}\times{\cal Y}^{\ell}\times{\cal U}^{*}\to{\cal X}^{\ell}$ is the output function, $g:{\cal Z}\times{\cal Y}^{\ell}\times{\cal U}^{*}\to{\cal Z}$ is the next–state function, and $\Delta:{\cal Z}\times{\cal Y}^{\ell}\to\{0,1,2,\ldots\}$. When \mbox{\boldmathu}=u_{1},u_{2},\ldots and the side information sequence, \mbox{\boldmathy}=y_{1},y_{2},\ldots, $y_{t}\in{\cal Y}$, $t=1,2,\ldots$, are fed into $Q$, it produces \hat{\mbox{\boldmathx}}^{n}, according to

[TABLE]

Note that here, we have somewhat generalized the model in the sense that the system is now fed by $\ell$–tuples of $y$ and it produces $\ell$–tuples of \hat{\mbox{\boldmathx}}. The reason is that in the context of systems with a side information input, input–output mechanisms that work on a symbol–by–symbol basis (i.e., $\ell=1$) are too limited. It is reasonable to allow dependencies between side information symbols and their corresponding output symbols with some delay and anticipation, and indeed, such a delay and anticipation will be needed in the achievability scheme. We could have allowed a general $\ell$ also in the earlier case, where no side information was available.

Let $G_{Q}(x^{n}|y^{n})$ denote the random number of guessing rounds needed for $Q$ until success. Next, for a given $\zeta>0$, define

[TABLE]

where ${\cal Q}(s,\ell)$ is the set of all FSGMs with block length less than or equal to $\ell$ and no more than $s$ states. For two given infinite sequences, \mbox{\boldmathx}=(x_{1},x_{2},\ldots) and \mbox{\boldmathy}=(y_{1},y_{2},\ldots), we define

[TABLE]

and finally,

[TABLE]

To present the results, we need a few more definitions. Consider the joint parsing of the sequence of pairs, $\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n})\}$, let $c(x^{n},y^{n})$ denote the number of phrases, $c(y^{n})$ – the number of distinct $y$-phrases, \mbox{\boldmathy}(j) – the $j$-th distinct $y$-phrase, $1\leq j\leq c(y^{n})$, and finally, let $c_{j}(x^{n}|y^{n})$ denote the number of times \mbox{\boldmathy}(j) appears as a phrase, or, equivalently, the number of distinct $x$-phrases that appear jointly with \mbox{\boldmathy}(j), so that $\sum_{j=1}^{c(y^{n})}c_{j}(x^{n}|y^{n})=c(x^{n},y^{n})$. Then, we define the conditional LZ complexity [13] as

[TABLE]

Let the conditional $K$–state compressibility of $x^{n}$ given $y^{n}$, denoted $\rho_{K}(x^{n}|y^{n})$, be defined as in [9, pp. 2245]: A $K$-state encoder $E$ with side information is defined by a set of six objects $(\Sigma,{\cal B},{\cal X},{\cal Y},p,q)$, where $\Sigma$ is a finite set of $K$ states, ${\cal B}$ is a finite set of binary words (possibly of different lengths, including the null word for idling), ${\cal X}$ is the finite alphabet of the source to be compressed, ${\cal Y}$ is a finite alphabet of side information, $p:\Sigma\times{\cal X}\times{\cal Y}\to{\cal B}$ is the encoder output function, and $q:\Sigma\times{\cal X}\times{\cal Y}\to\Sigma$ is the next–state function. When an input sequence $x_{1},x_{2},\ldots$ and a side information sequence $y_{1},y_{2},\ldots$ are fed together, sequentially into $E$, the encoder outputs a sequence of binary words $b_{1},b_{2},\ldots$, $b_{i}\in{\cal B}$, according to

[TABLE]

where $\sigma_{i}$ is the state of $E$ at time instant $i$. The decoder, on the other hand, receives the pair sequence $(b_{1},y_{1}),(b_{2},y_{2}),\ldots$ and reconstructs the source sequence $x_{1},x_{2},\ldots$. A finite–state encoder $E$ with side information is said to be information lossless (IL) if for all $\sigma_{1}\in\Sigma$ and all $(x^{n},y^{n})\in{\cal X}^{n}\times{\cal Y}^{n}$, $n\geq 1$, the quadruple $(\sigma_{1},\sigma_{n+1},b^{n},y^{n})$ uniquely determines $x^{n}$, The length function associated with $E$ is defined as $L_{E}(x^{n}|y^{n})=\sum_{i=1}^{n}l(b_{i})$, where $l(b_{i})$ is the length of the binary string $b_{i}\in{\cal B}$. We now define

[TABLE]

where ${\cal E}(K)$ is the class of IL encoders with no more than $K$ states. As shown in [9, eq. (13)],

[TABLE]

5.1 Converse Bounds

The converse bounds are as follows.

[TABLE]

where now $K(\ell)$ is redefined as $([\alpha\beta]^{k+1}-1)/(\alpha\beta-1)$ and $\delta_{n}(s,\ell)$ tends to zero uniformly as $n\to\infty$ for any fixed $(s,\ell)$. Note that now there is no maximization over all values of $\ell$ that are divisors of $n$ because now $\ell$ is a parameter of the model and not an auxiliary parameter as before (indeed, $\gamma_{s,\ell}(x^{n}|y^{n})\gamma_{s,\ell}(x^{n}|y^{n})$ at the l.h.s. depends on $\ell$ too).

The proof follows the same lines as before with a just few twists. Any FSGM $Q$ with $s$ states incudes a channel from $y^{n}$ to $x^{n}$ with the following structure,

[TABLE]

As in the earlier derivation, we have

[TABLE]

Consider now the partitioning $\hat{x}^{n}$ and $y^{n}$ into $m=n/\ell$ non–overlapping segments of length $\ell$. Then, once again,

[TABLE]

where $\hat{H}_{\ell}(x^{n}|y^{n})$ is the conditional empirical entropy of $\ell$–blocks. Similarly as in the earlier derivation, we can further lower bound the r.h.s. in terms of the conditional compressibility.

[TABLE]

and combining this with eq. (58), we obtain

[TABLE]

which, together with eq. (57), proves the first converse bound, and the second lower bound follows from [9]

[TABLE]

It follows that the second converse bound holds with

[TABLE]

5.2 Achievability

Following the same steps as in Section 4 and in [10], consider randomly drawing guesses according to the distribution

[TABLE]

where $LZ(\hat{x}^{n}|y^{n})$ is the length of compressed version of $\hat{x}^{n}$ given $y^{n}$ using the conditional version of the LZ78 algorithm [13, p. 460] (see also [12]). It is easy to see that this randomized guessing distribution asymptotically achieves the second lower bound, since

[TABLE]

where $\epsilon_{1}(n)$ is of the order of $\log(\log n)/(\log n)$, as shown in [13, p. 460]. Once again, to devise a matching direct in the framework of finite–state machines, we can restart every $\ell$–block and apply the random guessing distribution

[TABLE]

We now need to show that $\sum_{i=0}^{n/\ell-1}u(\hat{x}_{\ell i+1}^{\ell i+\ell},y_{\ell i+1}^{\ell i+\ell})$ is essentially no larger than $\rho_{K}(\hat{x}^{n}|y^{n})$ for some $K$ that can be chosen arbitrarily large, provided that $n$ is large enough. Once again, consider the following chain of inequalities for a given positive integer $K$:

[TABLE]

Thus,

[TABLE]

and the remaining steps are similarly as before.