Kolmogorov complexity in the USSR (1975--1982): isolation and its end

TL;DR

This paper discusses the history and decline of algorithmic information theory research in the USSR during 1975-1982, highlighting key figures and the eventual revival of interest.

Contribution

It provides a historical account of the USSR's isolation in algorithmic information theory and the factors leading to its end, emphasizing the role of Kolmogorov and Levin.

Findings

Interest in algorithmic information theory declined after 1970s

Levin's seminal contributions shaped modern theory

Kolmogorov's seminar sparked new research efforts

Abstract

These reminiscences are about the "dark ages" of algorithmic information theory in the USSR. After a great interest in this topic in 1960s and the beginning of 1970s the number of people working in this area in the USSR decreased significantly. At that time L.A. Levin published a bunch of papers that were seminal for the modern algorithmic information theory. Then he left the USSR, and the new wave of interest was triggered by the talk of A.N. Kolmogorov at a Moscow State (Lomonosov) University Mathematical Department (Logic and Algorithms Division) seminar organized by him; several younger researchers obtained some new results in algorithmic information theory.

Equations

$$\mathbf{c}\cup \mathbf{r}\cup \bigcup _{i=1}^{\infty }{\mathbf{a}}_i\cup \mathbf{d}=1,$$

Full text

1 Leonid Levin

I met Leonid Levin during the academic year 1971/2. At that time I was a first-year graduate student and lived (as well as many of them) in a dormitory called “The Graduate Students and Interns House” (Dom aspiranta i stazhera). This was a large dormitory; each room was occupied by four or five persons The inhabitants were a mixture of graduate students from all the departments of the Moscow State University, interns, visitors and many other strangers.

Leonid (Lenya) Levin graduated from the Moscow State University mathematics department (“mekh-mat”) a year before myself, in 1970. He was not accepted to the graduate school despite the support of Andrei Kolmogorov and despite of his obvious talents and achievements (e.g., he already published a survey [8] about Kolmogorov complexity, with many new results, together with Alexander Zvonkin). This was impossible due to the veto of the Russian secret service, KGB, and the communist party commitee (because of political reasons; moreover, as if it were not enough, Levin was of Jewish origin). He managed to get some temporary positions due to the help of Kolmogorov (if I remember correctly, Kolmogorov managed to obtain an intern position in his laboratory for Levin) and others (e.g., the vice-director of the Institute of Information Transmission Problems, Iosif Abramovich Ovseevich, secured later a temporary position in the Institute for Levin). From time to time one could see Lenya waiting in the queue of visitors to the dormitory manager when he tried to extend his official stay in the dormitory. On the other hand, one could illegally live there with some friends (if one has enough of them).

Lenya was acquainted, it seems, with all the graduate students of the math department (and some students from other departments) who lived in this dormitory, including myself. Everybody knew him, and we all called him Lenya. His had extremely wide interests at the time. It was difficult to find some topic in science (and not only in science) that was of no interest to him, and he had a strong opinion on most topics. He discussed virtually all mathematical results obtained at the time.

Moreover, often people from different applied fields came to Lenya and discussed their work. This (volunteer) consulting sometimes provided solutions for their problems.

At that time we never discussed Kolmogorov complexity. But I remember that Lenya told me about his (strong) results connected with step-counting functions, Rabin’s compression theorem and Blum’s speedup theorem. These were a hot topic at the time. These results were published together with the Russian translation of one of Blum’s papers [16]. He told me about these results.

The topic of my graduate work (1971–1974) was the semilattice of computable numbering of recursively enumerable sets. (Now these sets are called ‘computably enumerable’.) The notion of a computable numbering was introduced by Kolmogorov, and his student of 50s, Vladimir Uspensky, was my thesis advisor. In 1970s this topic was intensively developed in Novosibirsk Institute of Mathematics. People there were interested in my work; it became a topic of several expository talks at their “Algebra and logic” seminar.

I did not learn about Kolmogorov complexity and algorithmic randomness before 1975. However, I remember being (in 1970 or 1971) in a rather crowded room; it was a seminar organized by Kolmogorov, and more than 30 people were there (except for Kolmogorov, I remember also Levin, Max Kanovich and Nikolai Petri). Probably this was a predecessor of the Kolmogorov complexity seminar organized by him later in the beginning of 1980s.

I remember Kolmogorov’s attitude at the time: he considered Levin as a colleague rather than a student or a young researcher. I remember an occasion when Kolmogorov and Levin discussed some problem that Levin was working on; this problem was just solved by somebody else. Kolmogorov said to Levin something like “we are mature enough mathematicians not to be disappointed when somebody else was the first to solve the problem”.

One more reminiscence: during the seminar Kolmogorov asked Levin whether he could give a talk about something. Levin answered (in his style) that yes, he is able to give a talk on any topic at any time and in any place.

My contacts with Levin continued in 1975. At that year I finished my thesis, defended it and was sent for obligatory work to Bashkir State University in Ufa (a center of Bashkiria region in the USSR; after graduation, Soviet students have to work for three years at the place chosen by the authorities, only later they were allowed to change the job). There were good mathematicians in Ufa, but they were mostly interested in mathematical analysis or differential equations, and I had no chances for scientific collaboration there.

I visited Moscow quite often, and once met with Levin at the Moscow State University. He invited me to the apartment he rented in Moscow. It was near the subway station called “Kakhovskaya”, near the street called “Chernomorsky bulvar” (boulevard).

At that time he got some better position in some applied research institute where he has enough free time to work on his own topics. He had some Soviet-reasonable salary and was able to rent an apartment in Moscow. At the same time his life was quite stressful. He had (and not without a reason!) a permanent feeling that at any moment the Soviet secret services could interfere and force him to be fired and sent out of Moscow.

Lenya liked visitors and was quite happy if they stayed with him for a while. I think that many people (of approximately the same age as him) remember the apartment he rented, his hospitality, nice and joyful people and interesting talks (often oppositional to the authorities, as it was then taken among the intelligentsia). Often foreign visitors joined the company. I remember a funny philosophical discussion of freedom and non-freedom in the USSR. The American guest, a woman named Judy, who visited the USSR several times and liked the joyful companies and discussions, noted that freedom is an internal feeling. Lenya replied with a (metaphorical?) story. Peter Gacs (his Hungarian friend and colleague since 1970s, now they work both in Boston University) comes to Moscow with very tasty hungarian sausage roll. Each day he cuts and eats a small piece of the roll. When the sausage is over, he goes back to Hungary (that was considered at the time as a bit more developed country than USSR).

During the summer of 1975 I came to Moscow for a long time, and during July we made long walks with Lenya. While walking in the Bitsa park (a rather large park that was on the boundary of Moscow at that time), we discussed almost everything: politics, philosophy, mathematics, universal prediction and universal semimeasure, the notion of information content for infinite sequences. Sometimes Albert Abramovich Muchnik (of Muchnik – Friedberg fame) joined our discussions.

Levin was a fast thinker and usually was ahead in the conversation and could convince other participants. However, it was easy to speak with him. He never tried to enforce his opinion upon the others. He just provided rational arguments that usually convinced others without pressing them to accept Levin’s viewpoint. This made him a popular person and many people had high regard for him. It seems that local party bosses and the KGB understood this and were suspicious about possible influence of Levin. People like him had no place in the USSR.

In 1975 Levin formulated a question that he considered as philosophically important. After some simplification, this problem was stated as follows. Consider infinite binary sequences modulo Turing reductions, i.e., Turing degrees. One can say that two mutually reducible sequences can be transformed one into another by a computable operators (oracle machines), and therefore “contain the same information” (up to the finite amount of information hidden in the reductions). We consider Borel sets (properties) of infinite sequences that are invariant under these reductions, i.e., unions of arbitrary families of Turing degrees. One of these invariant families is formed by (Turing degrees) of Martin-Löf random sequences. Levin showed earlier that one can consider only sequences that are random with respect to the uniform Bernoulli measure: every sequence that is Martin-Löf random with respect to some computable measure, has the same degree as some sequence that is Martin-Löf random with respect to the uniform measure [8]. The only exceptions are computable sequences. They are random with respect to computable measures where they are atoms (have positive measure), but are not Turing-equivalent to sequences that are Martin-Löf random with respect to the uniform measure (those sequences cannot be computable for obvious reasons). The computable sequences form another invariant class. Now the question is whether they exist other non-trivial invariant classes.

Of course, this question should be formulated more precisely to be interesting. Note that the union of an arbitrary family of Turing degrees is an invariant class (if measureble). For example, one may consider just one non-computable Turing degree, i.e., the set of all sequences that are Turing-equivalent to a given non-computable sequence. However, such a property is not “real” in a sense that no probabilistic process (algorithm with access to a fair coin) producing a finite or infinite binary sequences can produce a sequence with this property (from a fixed non-computable Turing degree) with positive probability. Indeed, the famous theorem of de Leuuw, Moore, Shannon and Shapiro [22] guarantees this. There are other invariant sets of sequences with the same property of being “non-real” (no probabilistic process can generate their elements with positive probability). Levin gave a description for classes with this property. Namely, for each set with this property there is some sequence $\alpha$ such that every sequence from the set contains infinite amount of information about $\alpha$. Moreover, for each $\alpha$ the set of sequences that contain infinite amount of information about $\alpha$, has this property (see [15]).111Levin gave some definitions of mutual information in two infinite sequences in this paper and subsequent ones. The quantity of information $I(\alpha:\beta)$ should have an invariance property: it cannot be increased by random or deterministic (recursive) processing of $\alpha$ or $\beta$ (see, e.g., [33]). He insisted that this notion is very important and different approaches to its definition should be developed. In this way we get invariant sets of sequences whose universal measure (maximal lower semicomputable continuous semimeasure, called also a priori probability) is zero.222 The semicomputable universal (a priori) measure was introduced in [8]. The a priori measure of a Borel set of infinite sequences is equal to the probability that the corresponding probabilistic Turing machine will produce a sequence from this set. Levin called sets with this property negligible.

We want to ignore negligible sets. Formally, consider the equivalence relation on invariant sets: two sets are equivalent if their symmetric difference is negligible. In this way we get a structure that was called the algebra of invariant properties. The operations in this algebra are restrictions of the usual set-theoretic operations to invariant sets. The minimal element is the equivalence class $\mathbf{0}$ of negligible sets. The maximal element is the class $\mathbf{1}$ of all sequences. There are two naturally defined elements in this algebra: the class $\mathbf{r}$ generated by Martin-Löf random sequences, and the class $\mathbf{c}$ generated by computable sequences.

Levin told me that the elements $\mathbf{c}$ and $\mathbf{r}$ are atoms of the algebra of invariant properties, i.e., cannot be represented as a union of two other non-zero elements from this algebra.

Levin [8, definition 4.4] called “proper” (this term is used in the English translation of that article; Russian name was <<правильная последовательность>>) or “complete” [32, 33] the sequences that are Martin-Löf random with respect to some computable measure. Sequences that are Turing-equivalent to complete sequences are called regular (see [33]).

Every regular sequence $\alpha$ is either computable or Turing-equivalent to some sequence $\beta$ that is random with respect to uniform Bernoulli measure (see [8]). The prefixes of $\beta$ have monotone complexity equal to their length (up to a constant). Therefore one could say that information contained in a regular sequence can be compressed maximally. The property of being atoms (for $\mathbf{c}$ and $\mathbf{r}$) can be interpreted as follows: the only invariant information about a regular sequence is the amount of information in it that can be either finite ($\mathbf{c}$) or infinite ($\mathbf{r}$). The information in the elements of $\mathbf{r}$, after being compressed optimally, is indistinguishable from random noise.

Levin then asked me: is it true that there are no other elements (except for the elements generated by computable and Martin-Löf random sequences, and, of course, $\mathbf{0}$ and $\mathbf{1}$) in this algebra, i.e., $\mathbf{c}\cup\mathbf{r}=\mathbf{1}$, or some other elements exist?

When I returned to Ufa from Moscow (in the summer of 1975), I managed to construct a counterexample to the conjecture that no other elements exist, i.e., a probabilistic algorithm whose output is with positive probability (that can be made close to $1$ but not equal to $1$) a noncomputable sequence that is not Turing-equivalent to any Martin-Löf random sequence. Therefore, $\mathbf{c}\cup\mathbf{r}\neq\mathbf{1}$ and the algebra contains more elements (not only the four elements mentioned), see [3].

From a philosophical viewpoint one could say that there is a natural process that with positive probability generates arrays of information that cannot be “explained” as being random with respect to some computable measure. One can even say that these non-regular sequences cannot be optimally compressed without infinite information loss. This can be considered as an application of computability theory to the foundations of probability theory (and foundations of mathematics in general).

As we already mentioned, Levin has proven that $\mathbf{c}$ and $\mathbf{r}$ are atoms in the algebra of invariant properties (cannot be presented as non-trivial unions of smaller elements). Levin then asked: what about other atoms in this algebra? It is easy to see that there is at most countable set of atoms. Approximately in 1977–1978 I managed to construct indeed a countable family of atoms, and an infinitely divisible element (that has no atoms as subsets). Therefore, we have the disjoint union

[TABLE]

where $\mathbf{a}_{i}$ are atoms that do not contain computable sequences, and $\mathbf{d}$ is the rest that is the infinitely divisible element. It seems that this result has no impact on further research, and it is a pity that the properties of these $\mathbf{a}_{i}$ are still not studied.

At that time Levin was developing a general theory of random objects. He wanted to define uniform randomness deficiency (with respect to semimeasures, i.e., measures on the set of finite and infinite sequences) in such a way that every infinite binary sequence would be random with respect to the maximal semimeasure (now also sometimes called “the continuous a priori probability”).333Some results in this direction were later published in [33]. He also dreamed about a universal language that would allow us to give concise definition for mathematical objects and functions. The hope was that by using this language one could make the hidden constants in the statements about Kolmogorov complexity reasonable. We discussed these questions again and again in 1975–1978.

Levin also told me about the “brute force search problem” (¡¡problema perebora¿¿ in Russian) that later become famous as $\text{P}=\text{NP}$ question. He stated the problem in a short note [13] published in “Information Transmission Problems” journal; this note contained several examples of NP-complete problems. It went almost unnoticed at that time, but later became rather famous. The history of the early attempts to approach this problem in the USSR is explained in an excellent survey by Boris Trakhtenbrot [36]. In the same paper Levin noted that there is an algorithm (sometimes called “L-search”) that is optimal in the sense that if there is an efficient (e.g., polynomial) algorithm for NP-complete problems, then Levin’s algorithm is almost as fast. (Of course, the catch is that nobody knows whether this algorithm is polynomial.) The argument that Levin had in mind used some new version of Kolmogorov complexity that takes into account not only the program size but also the (logarithm) of its computational time. However, the paper was rather short (as usual for Levin), and the result was stated not only without proof, but also without the description of the search algorithm. That time Levin told me about this algorithm and the corresponding complexity.444Later this algorithm was presented in [33], see also the survey [38].

Levin suggested that I should work on $\text{P}=\text{NP}$ question: at that time the situation does not look as hopeless as now. Probably it was a good luck for me that my personal life has distracted me from concentrating on this problem.

In 1978 Levin decided to emigrate to the US using the Israel invitation as a tool (as many people did: getting an invitation to repatriate to Israel was an unreliable but essentially the only way to get out of the USSR). It is remarkable that Levin waited so long. This way out of the USSR was possible (at least for some people, mostly Jews) already for several years, and the KGB people who summoned Levin gave advices to leave Moscow, e.g., to go to Armenia (that was at that time a part of the USSR). This could be a hint that Levin emigration would be desirable for KGB and he would be allowed to leave.

At that time Soviet border, like Styx, was essentially one-way. The emigrant should renounce the Soviet citizenship (i.e., write a letter asking for that; during some periods, there was a quite significant fee for such a procedure). Of course, there were other problems, like language barrier. We both knew enough English to read a mathematical paper almost without a dictionary, but Levin’s knowledge of Russian language and culture was magnificent: for example, he knew a lot about Alexander Blok, Russian poet (“Silver Age” of Russian culture, the beginning of XX century) and could read by heart his poems for hours.

During our cooperation in 1975–1978 Lenya was quite isolated in a scientific sense; he cannot go to conferences and seminars abroad (the authorities definitely would not let him out, so he did not even try). His PhD thesis (under the supervision of Kolmogorov) was failed in Novosibirsk despite of the positive reports of all the reviewer and the advisor (Kolmogorov). (See more details in [25]. The translated version of this thesis was published in [31].)

Once Levin was invited to a computer science conference (Mathematical Foundations of Computer Science) in Czechslovakia; at that time it was a Soviet-controlled country (so authorities could not worry about a chance to escape) but nevertheless none of us could go there officially. (The invitation was sent to Levin by some member of the organizing committee. He suggested that I could try to go there, but my Ufa bosses told me that I have no chance — one should get 18 different kinds of permissions for that.) Still we sent a paper [32] there, though it was forbidden at the time (one has to apply for a special permission to send the paper abroad; for that a special official certificate was needed that confirms that “the paper does not contain anything that could be considered as a discovery or an invention”). This well could end in firing from the job (or even worse), but the Soviet system was quite old at the time and couldn’t keep track of all violations of that (relatively small) scale.

Of course, Levin has many friends with whom he could discuss scientific questions, including already mentioned Alexander Zvonkin (the coauthor of the survey [8]), Albert Muchnik, Peter Gács. And, of course, Levin was always supported by Kolmogorov. As an academician, Kolmogorov was able to endorse Levin’s publications in the Reports of the Academy of Sciences journal (“Doklady AN SSSR”), and quite a few Levin’s papers were published there. Levin was also able to publish papers in another reputable journal, Information Transmission Problems, published by the IITP (Institute for Information Transmission Problems).

However, this was more a hobby than a regular activity in the framework of some academic or research institution. The problems of the algorithmic information theory were not discussed at the university, in particular, Levin could not be a student supervisor. The algorithmic information theory was “a marginal” science.

It is not a coincidence that Levin published more during a difficult periods of his life. Here are some papers that cover his main results in the algorithmic information theory: [14, 15, 17, 18, 19, 20, 21].

It seems I was the only collaborator of Levin from the USSR working with him on algorithmic information theory. But Peter Gacs that came several times from Hungary worked with Leonid and obtained significant results in the field.

Looking back, I feel that I was lucky that Levin has so much time to discuss mathematics (and other topics) with me. Were he a prominent and recognized scientist in the USSR (at the level of his international recognition later), he would be rather busy and we would have much shorter and much more formal meetings.

Trying to follow Judy’s example and find some advantages of Soviet life, I could say that the management of our official jobs did not care much about what we are doing and often allowed us to do what we wanted not caring about the results, there was no pressure to get papers published, to write grant applications and reports, etc. Having few career opportunities, we could be “mentally free people” and could work for a long time on some abstract (almost philosophical) problems — or just relax and do nothing.

In 1977 I got married and was allowed to come to Moscow. Looking for a job in Moscow was a hard task. It was impossible to get a teaching position in some higher education institution: there were many qualified people who did not want to lose their right to live in Moscow by going elsewhere. The only jobs available were the so-called “otraslevye instituty”, centers organized by government to work on the problems of specific industry. I changed three jobs of this type. In the first and the second ones I was able to work only for a year, or, better to say, to be paid. The “work” there was completely disorganized, people did not do anything (and often knew nothing about the industry they were assumed to help), the management gave idiotic orders, etc. On the other hand, people with PhD, including myself, were given a good (according to Soviet standards) salary. At that time I tried to ignore the surrounding farce and finish my paper about invariant properties.

In 1978 Lenya and his wife Larissa got a permission to emigrate and went to Vienna by a train (from the “Belorusskaya” railroad station). Many people came to this station to say goodbye to them. When leaving, Lenya told me that he believes that algorithmic information theory will get more attention in the USSR. He also wrote a reference and gave a specific advice: to show my paper about the algebra of invariant properties to Kolmogorov, and gave me his phone number.

Leonid Levin (3rd from the left) and his wife Larissa (8th from the left) and their friends just before their departure from the USSR (also Eduard Dumanis (5) and his wife (4), Vladimir V’yugin (5) and his wife Elena (7), Albert Muchnik (9))

2 Andrei Nikolaevich Kolmogorov

When my paper was ready, approximately in 1979, I phoned Kolmogorov and said that Lenya Levin suggested that I show my paper on the algorithmic information theory to him (Kolmogorov). Kolmogorov gave me his address (he lived in the building of the Moscow State University, near the department of mathematics) and told me when to come. When I arrived, he opened the door, asked who I am and took my paper. He asked me to call him again in two weeks. When I did that, he started to apologize that he has no time to read the paper, and asked me to call again in two weeks. The same thing repeated for a long time, it took may be a year and a half. Of course, sometimes I waited longer than two weeks, sometimes Kolmogorov was ill for few months, etc.

Levin was right predicting that interest in Kolmogorov complexity would revive. My thesis advisor, Vladimir Uspensky, asked me (around 1980) to write a survey about Kolmogorov complexity and algorithmic randomness. I never tried to approach systematically the notions of algorithmic information theory, but I remembered our discussions with Levin (while walking), also his papers of 1973–1977 (mostly in Soviet Math. Doklady) were accessible. Also I started to read papers of Chaitin [26, 27, 28, 29] and others, starting with a seminal paper of Kolmogorov [9] (in Information Transmission Problems). Following the path of history, one should start with the definition of plain Kolmogorov complexity and the notion of randomness deficiency for a finite sequence in a finite set. (My opinion is that frequency definitions of randomness are now obsolete and are interesting only from the historical viewpoint.) All this can be called Kolmogorov’s approach to the definition of a random finite object. Then one should give a definition of randomness given by Martin-Löf for infinite sequences, and show that a difficulty arises when we try to characterize randomness of the sequence in terms of the plain complexity of its prefixes (Martin-Löf’s result saying that every sequence has prefixes where difference between length and complexity is large). The next step was the prefix and monotone complexities as invented by Levin, the theorem of Levin that characterizes randomness in terms of monotone or prefix complexity.555The prefix complexity and its application to the definition of randomness also was invented by Schnorr and Chaitin. Schnorr proposed two different definitions of monotone complexity using monotone functions and monotone operators, see details in [4]. Finally, one should introduce the notion of a Levin universal semimeasure and the corresponding criterion of randomness.

This was my plan; I had to reconstruct most of the proofs since they often were missing in original Levin’s papers. It would be nice to have some general framework for different flavors of Kolmogorov complexity, but I did not succeed in doing this. While writing the paper, I got some useful feedback from Sasha Shen (whom I met then for the first time). Later (in his PhD thesis) he managed to provide a general scheme for different versions of complexity in terms of computable continuous mappings (that characterization was translated to a more accessible language later by Uspensky and Shen).

This survey was published in the series of volumes called Semiotics and Informatics in 1981 [4]. Li and Vitanyi [34, page 193] wrote that “another source for the Russian school is the survey by V.V. Vyugin, Selecta Mathematica, formerly Sovietica, 13:4 (1994), 357–389 (translated from the Russian Semiotika and Informatika, 16 (1981), 14–43).” For Russian readers it was the second available survey paper after [8], and it included main Levin’s results about randomness with proofs.

The experienced people explained me how the Levin’s papers could be cited still keeping the paper publisheable: references to his papers are more or less OK, assuming that name ‘Levin’ never appears in the text of the paper, only in the list of references. (In particular, I couldn’t express my gratitude to him in the text.)

When I called Kolmogorov next time (about September 1980), he said that he has read my paper and finds it interesting. However, he does not agree completely with the interpretation of the results. He said also that he would like to write a commentary and later entire paper that explains his views, and it would be nice to publish these papers together in one volume of some journal. He said also that he has read my survey and likes it (and later he even made a reference to it in his paper [12]).

Kolmogorov invited me to his apartment and typed (while I was there) his commentary, using his electric typewriter. At that time he was more and more struggling with the Parkinson disease and it was difficult for him to type, several letters were typed repeatedly. So I went to the office of the Logic Division of the Mathematics Department of Moscow State University to retype the commentary. I still keep the copy of the Kolmogorov’s typescript.

Kolmogorov never wrote an article he planned, and I submitted my article to the Information Transmission Problems journal. The editorial office told me that they never accept any commentaries together with the paper. The paper was published in 1982 as [5].

After our meeting Kolmogorov gave a talk at the seminar that was organized by him, Alexei Semenov and Alexander Shen, called “Definition complexity and computational complexity” (Slozhnost’ opredelenii i slozhnost’ vychislenii). One could say that this was a continuation of a seminar led by Kolmorogov and Levin many years earlier.

In his talk Kolmogorov gave the definition of an $(\alpha,\beta)$-stochastic sequence. As people remember, Kolmgorov had the similar notions much earlier, even before 1974 (see [30]). I guess that my phone calls reminded him of this notion and he suggested to study its properties. Now this approach is developed as a part of the so-called “algorithmic statistics” (see [37]).

I would explain Kolmogorov’s definition as follows. Fix arbitrary non-negative numbers $\alpha$ and $\beta$. A string $x$ is called $(\alpha,\beta)$-stochastic if it is a “$\beta$-random” element of some “$\alpha$-simple” sets — more precisely, if $x\in A$ and ${\rm K}(x\mskip 1.0mu|\mskip 1.0muA)\geqslant\log|A|-\beta$ for some finite set $A$ of strings with ${\rm K}(A)\leqslant\alpha$. Here $|A|$ is the cardinality of $A$, and ${\rm K}(x\mskip 1.0mu|\mskip 1.0muA)$ is the conditional Kolmogorov complexity of $x$ if $A$ is given (by list of its elements), and all logarithms are binary.

If $x$ is not $(\alpha,\beta)$-stochastic, this means that there is no simple model that “explains” $x$ is statistical sense.

Kolmogorov asked in his talk whether non-stochastic strings exist (for reasonable values of $\alpha$ and $\beta$) and, if yes, how many of them can be found among strings of given length. For such a sequence we have $d(x\mskip 1.0mu|\mskip 1.0muA)=\log|A|-{\rm K}(x\mskip 1.0mu|\mskip 1.0muA)>\beta$ for every finite set $A$ containing $x$ such that ${\rm K}(A)\leqslant\alpha$. Let us call $d(x\mskip 1.0mu|\mskip 1.0muA)$ the randomness deficiency of $x\in A$ with respect to a finite set $A$ see [11]).

Later A. Shen [24] answered Kolmogorov’s question in its simplest version: he gave conditions on $\alpha$ and $\beta$ that guarantee the existence of non-stochastic sequences and proved that the fraction of nonstochastic sequences among all strings of given length decreases exponentially as $\alpha$ and $\beta$ increase.666 In 1998 Levin told me that he had discussed all these issues with Kolmogorov in the early 1970s and and got the relevant results but not published them.

I thought it would be important to consider not only the fraction of non-stochastic sequences (i.e., their measure according to the uniform distribution), but the probability to get a non-stochastic sequence as an output of some randomized algorithm. Formally speaking, one should find the a priori probability of the set of all non-stochastic strings of given length.777More correctly, we consider the a priory measure of all continuations of strings from this set. This value is equal to the probability with which a probabilistic Turing machine can print an $(\alpha,\beta)$-non-stochastic sequence (or its continuation).

Working in this direction, I established in [6] some lower and upper bounds for the a priori probability of the set of all $(\alpha,\beta)$-nonstochastic strings of length $n$ (in terms of $\alpha$ and $\beta$).

An asymptotic version of these results can be stated as follows. Consider the a priori probability of the set of all $(\alpha,\beta)$-nonstochastic strings of length $n$. Here $\alpha=\alpha(n)$ and $\beta=\beta(n)$ are considered as functions of $n$. This measure decreases as $2^{-\alpha(n)}$ if $\alpha(n)$ and $\beta(n)$ are unbounded computable functions. However, if we do not require the functions $\alpha(n)$ and $\beta(n)$ to be computable, then (as shown in [6]) one can find two non-decreasing unboundeds function $\alpha(n)$ and $\beta(n)$ and a probabilistic machine that generates with positive (or even close to $1$) probability an infinite sequence whose $n$-bit prefixes are not $(\alpha(n),\beta(n))$-stochastic for infinitely many $n$.888Here the function $\alpha(n)$ can be made upper semicomputable and the function $\beta(n)$ can be made computable.

It is easy to see that infinite sequences with this property cannot be Martin-Löf random with respect to any computable measure. One can modify the argument to guarantee that machine generates (with positive probability) sequences that have this property and at the same time are not Turing-equivalent to Martin-Löf random sequences.

One could say that the philosophical question about “real existence” of non-stochastic objects could have different answers depending on its interpretation. If we fix some computable bounds $\alpha(n)$ and $\beta(n)$ for non-stochasticity, the a priori probability of the set of all non-stochastic sequences decreases exponentially. On the other hand, by choosing non-computable bounds, we may generate (with probability close to $1$) infinite sequences who have infinitely many non-stochastic prefixes.

It seems to me that Kolmogorov and Levin have different philosophical views on what should be the subject of algorithmic randomness theory. Kolmogorov thought that the most important notion of randomness deals with finite objects (binary strings), and should not use the notion of measure (probability distribution). He believed that one should derive the properties of a string from the assumption that this string has maximal complexity in some class of strings [12]. More precisely, consider some equivalence relation on the set of $n$-bit strings. This relation defines equivalence classes; $n$-bit string $x$ is considered as $m$-random with respect to a partition if ${\rm K}(x\mskip 1.0mu|\mskip 1.0muD)\geqslant\log|D|-m$ where $D$ is the equivalence class containing $x$ (it is used as a condition in the conditional complexity of $x$), and $|D|$ denotes its cardinality. Encoding objects by their ordinal numbers, we get ${\rm K}(x\mskip 1.0mu|\mskip 1.0muD)\leqslant\log|D|+c$ where $c$ is some constant.

As an example, the definition of the $m$-Bernoulli sequence given by Kolmogorov in [10] can serve. A finite sequence $x$ is called $m$-Bernoulli if $K(x|n,k)\geqslant\log{n\choose k}-m$, where $n$ is the length of this sequence and $k$ is number of ones. The element of the corresponding partition consists of all sequences of a given length and with a given number of ones.

This approach was developed by Eugene Asarin (one of the last Kolmogorov’s students), its main findings are presented in his unpublished thesis (see also [1]).

Levin believed that the algorithmic randomness theory should be applicable both to finite and infinite objects, and the notion of randomness depends on the assumed measure, so measure should be used. Therefore he used monotone complexity that is better suited to the definition of randomness for infinite sequences. For the same reason he considered infinite sequences and their degrees when discussing non-stochasticity.

It is possible that this philosophical difference (what is more important: finite or infinite objects) was meant by Kolmogorov when he said that he disagrees with the interpretation of results from my paper. And, indeed, he always considered finite objects when speaking about non-stochasticity.

After Kolmogorov’s talk mentioned above the interest in algorithmic information theory revived in the USSR. Kolmogorov, Uspensky and Semenov had several good students that worked in this field, including Asarin, Vladimir Vovk, Shen, Andrei Muchnik. Uspensky, Semenov and Shen wrote a survey [23]; recently a monograph [2] appeared. But this period of the development of algorithmic information theory in the USSR and later Russia is beyond this paper.

The author is grateful to Alexander Shen for useful discussions that led to the clarification of many concepts and results.