A Variation of Levin Search for All Well-Defined Problems

TL;DR

This paper presents a simplified variation of Levin Search, inspired by Hutter's work, that efficiently solves well-defined problems by leveraging provable bounds on the fastest algorithms.

Contribution

The paper introduces a new, simpler algorithm that improves upon previous methods by reducing the runtime bound to a quadratic function of the proof length times the fastest algorithm's runtime.

Findings

Achieves problem-solving time bounded by O(l(f)^2) times the fastest algorithm's runtime.

Simplifies the existing algorithms by integrating ideas from Hutter's approach.

Provides a more practical and theoretically efficient method for solving well-defined problems.

Abstract

In 1973, L.A. Levin published an algorithm that solves any inversion problem $\pi$ as quickly as the fastest algorithm $p^*$ computing a solution for $\pi$ in time bounded by $2^{l(p^*)}.t^*$, where $l(p^*)$ is the length of the binary encoding of $p^*$, and $t^*$ is the runtime of $p^*$ plus the time to verify its correctness. In 2002, M. Hutter published an algorithm that solves any well-defined problem $\pi$ as quickly as the fastest algorithm $p^*$ computing a solution for $\pi$ in time bounded by $5.t_{p}(x)+d_p.time_{t_{p}}(x)+c_p$, where $d_p=40.2^{l(p)+l(t_{p})}$ and $c_p=40.2^{l(f)+1}.O(l(f)^2)$, where $l(f)$ is the length of the binary encoding of a proof $f$ that produces a pair $(p,t_p)$, where $t_p(x)$ is a provable time bound on the runtime of the fastest program $p$ provably equivalent to $p^*$. In this paper, we rewrite Levin Search using the ideas of Hutter so that we have a new simple algorithm that solves any well-defined problem $\pi$ as quickly as the fastest algorithm $p^*$ computing a solution for $\pi$ in time bounded by $O(l(f)^2).t_p(x)$.

Equations

$$K_t(x)=\min {.}_p\{l(p)+\log t:U(p)=x\text{, in at most }t\text{ steps.}\}$$

$$tim{e}_{LevinSearch}\le 2+4+\dots +\frac{T}{4}+\frac{T}{2}+T=2T-2<2T.$$

$$t^{\ast }=T.2^{-l(p^{\ast })}>\frac{tim{e}_{LevinSearch}}{2}.2^{-l(p^{\ast }).}$$

$$tim{e}_{LevinSearch}<2^{l(p^{\ast })+1}.t^{\ast }=O(t^{\ast })$$

$$tim{e}_{Hutter}(x)\le 5.t_{p_{fast}}(x)+d_p.tim{e}_{t_{p_{fast}}}(x)+c_p,$$

$$t_1$$

$$\le t+\sum _{f:l(f)\le \log t}O(l(f))+O(l(f{)}^2)$$

$$t_1$$

$$<t+2^{\log t+1}.O(l(f^{\prime }{)}^2)$$

$$<t+2t.O(l(f^{\prime }{)}^2)$$

$$<O(l(f^{\prime }{)}^2).t,$$

$$tim{e}_{ModifiedLevinSearch}$$

$$=\sum _{t=2}O(l(f^{\prime }{)}^2).t$$

$$<\sum _{t=2}O(l(f^{\ast }{)}^2).t,$$

$$tim{e}_{ModifiedLevinSearch}(x)$$

$$=O(l(f^{\ast }{)}^2).(2T-2)$$

$$<O(l(f^{\ast }{)}^2).2T$$

$$=O(l(f^{\ast }{)}^2).T$$

$$\le O(l(f^{\ast }{)}^2).t_{p_{fast}}(x).$$

$$tim{e}_{Hutter}(x)$$

$$\le 5.tim{e}_p(x)+d_p.tim{e}_p(x)+c_p$$

$$\le (5+d_p).tim{e}_p(x)+c_p$$

$$\le (5+40.2^{l(p)}).tim{e}_p(x)+40.2^{l(f)+1}.O(l(f{)}^2)$$

$$tim{e}_{ModifiedLevinSearch}$$

$$tim{e}_{ModifiedLevinSearch}<O(l(f{)}^2).2^{l(p)}.tim{e}_p(x)$$

Full text

keywords: Computational Complexity; Algorithmic Information Theory; Levin Search.

1 Introduction

We recall that the class $NP$ is the set of all decision problems that can be solved efficiently on a nondeterministic Turing Machine. Alternatively, $NP$ is the set of all decision problems whose guessed solutions can be verified efficienlty on a deterministic Turing Machine, or equivalently today’s computers. The interesting part is that the set of all hardest problems in the class $NP$ (also known as $NP$-complete problems) have so far defied any efficient solutions on real computers. This has been a very frustrating challenge given that many of the $NP$-complete problems are about real-world applications that we really would like to be able to solve efficiently.

Leonid Levin [6], and independently from Stephen Cook [1], defined the class $NP$, identified the class $NP$-complete, and gave a few reductions that show the $NP$-completeness of some $NP$ problems (independent from the work of Richard Karp [3]). In that same paper [6], Levin gave an algorithm to deal with $NP$-complete problems provided that someone develops a proof that shows $P=NP$. In this case, Levin’s algorithm can be used to develop a polynomial-time solution for every $NP$-complete problem. In particular, Levin’s algorithm works with a very broad class of mathematical problems that can be put in the form of inverting easily computable functions. For example, suppose we have a function $y=f(x)$, where we know nothing about $f$ except that it is easily computable. The challenge is to find $x=f^{-1}(y)$ and to do so in a minimal amount of time. The idea of Levin’s algorithm is to go through the space of all algorithms in search for a fastest algorithm $A$ that knows the secret of $f$. In general, let $\pi$ be an inversion problem and let $p^{*}$ be a known fastest algorithm for $\pi$ that runs in time $t^{*}$, where $t^{*}$ is the runtime of $p^{*}$ plus the time to verify its correctness. The Universal Search algorithm of Levin, also known as Levin Search, is an effective procedure for findng $p^{*}$ in time bounded by $2^{l(p^{*})}.t^{*}$, where $l(p^{*})$ is the length of the binary encoding of $p^{*}$. Levin Search can also be used with some forms of optimization and prediction problems, and it is theoretically optimal. The following is a pseudocode for Levin Search: **Pseudocode Levin Search

$t=2$**.

for all programs $p$, in parallel, do

run $p$ for at most $t.2^{-l(p)}$ steps

if $p$ is proved to generate a correct solution for $\pi$, then return $p$ for $p^{*}$ and halt.

endfor

$t=2.t$

goto for all programs $p$

**End of Pseudocode

**The idea of Levin Search is to search the space of all programs $p$ in increasing order of $l(p)+\log t$, or equivalently in decreasing order of $\frac{Prob.(p)}{t}$, where $Prob.(p)=2^{-l(p)}$ is the probability contributed to $p$ in the overall Solomonoff’s algorithmic probability [8] of finding a solution for $\pi$. By construction, $Prob.(p)$ gives short programs a better chance of being successful on the assumption that short programs are more worthy (Occam’s razor). We mention that the main idea of Levin Search was behind the notion of Levin Complexity, which defines a computable version of Kolmogorov complexity based on time-bounded Turing machines. Let $U$ be a fixed reference universal Turing machine, the Levin complexity of a finite binary string $x$ is defined as

[TABLE]

Returning to Levin Search, during each iteration of the “for” loop, each program is assigned a fraction of time proportional to its probability, which basically says that short programs get to run more often.

The rest of this paper is organized as follows. Section 2 describes a sequential implementation of Levin Search and shows that it solves any inversion problem $\pi$ as quickly as the fastest algorithm $p^{*}$ computing a solution for $\pi$, save for a factor of $2^{l(p^{*})}$. Section 3 describes an improvement on Levin Search for all well-defined problems. A problem is designated as well-defined if its solutions can be formally proved correct and have provably quickly computable time bounds. This improvement, known as Hutter’s algorithm [2], describes an algorithm that solves any well-defined problem $\pi$ as quickly as the fastest algorithm computing a solution for $\pi$, save for a factor of $5$. Section 4 contains our contribution. We rewrite Levin Search using the ideas of Hutter so that we have a new simple algorithm that solves any well-defined problem $\pi$ as quickly as the fastest algorithm $p^{*}$ computing a solution for $\pi$, save for a factor of $O(l(f)^{2})$, where $l(f)$ is the length of the binary encoding of a proof $f$ that produces a pair $(p,t_{p})$, where $t_{p}$ is a provable time bound on the runtime of the fastest program $p$ provably equivalent to $p^{*}$. Section 5 contains some concluding remarks.

2 Sequential Levin Search

The pseudocode of Levin Search from the previous section cannot be readily executed on today’s computers. A sequential version of Levin Search can simulate the parallel execution of all programs by running programs, each for a fraction of time, one after another, in increasing order of their length. Hereafter, we assume that all progams are encoded using prefix-free codes, using for example, Shannon-Fano coding [9]. Let $\pi$ be an inversion problem and let $p^{*}$ be a known fastest algorithm for $\pi$ that runs in time $t^{*}$, where $t^{*}$ is the runtime of $p^{*}$ plus the time to verify its correctness. Also, let $l(p)$ be the length of the binary encoding of program $p$. **Algorithm Levin Search

$t=2$**;

for all programs $p$ of length $l(p)=1,2,\ldots,\log t$ do

run $p$ for at most $t.2^{-l(p)}$ steps.

if $p$ is proved to generate a correct solution for $p$, return $p$ for $p^{*}$ and halt.

endfor

$t=2.t$

goto for all programs $p$

End of Levin Search During each time step $t$, the algorithm runs all programs $p$ of length $\leq\log t$ in increasing order of their length and gives each program $p$ a fraction of time proportional to $2^{-l(p)}$. Time Analysis. The following proof for the runtime complexity of Levin Search is due to Solomonoff [7]. The “for” loop in the algorithm takes time $\sum\limits_{p:l(p)\leq\log t}t.2^{-l(p)}\leq t$. This is true by Kraft inequality [5]. Now, suppose that Levin Search stops at time $t=T$. Then, the total runtime of this algorithm is

[TABLE]

Moreover, the runtime $t^{*}$ of $p^{*}$ is

[TABLE]

Thus,

[TABLE]

This is true because the multiplicative constant $2^{l(p^{*})+1}$ is independent of the instance of the problem $\pi$. We mention that it is this huge multiplicative constant that limits the applicability of Levin Search in practice.

It seems that a possible way to improve the runtime of Levin Search is to find a way, where not all programs of length $\leq\log t$ get executed, for each value of $t$. Ideally, we would like a way that executes at most a single program for each value of $t$, preferably a shortest and fastest one, and this is exactly what we will present in Section 4.

3 Hutter’s Fastest and Shortest Algorithm for All Well-Defined Problems

In [2], Hutter presented an improvement on Levin Search for problems that are well-defined. By this, Hutter means a problem whose solutions can be formally proved correct and have provably quickly computable time bounds. Let $p^{*}(x)$ be a known fastest algorithm for some well-defined problem $\pi(x)$. Then, Hutter’s algorithm will construct a solution $p(x)$ that is provably equivalent to $p^{*}(x)$, for all $x$, in time proportional to $5.t_{p}(x)+d.time_{t_{p}}(x)+c$, where $t_{p}(x)$ is a provable time bound on the runtime of $p(x)$, $time_{t_{p}}(x)$ is the time needed to compute $t_{p}(x)$, and $d$ and $c$ are constants which depend on $p$ but not on $\pi(x)$. This shows that Hutter’s agorithm runs in time $O(t_{p})$, save for a factor of 5.

Our work in the next section is inspired by the ideas of Hutter, and so, for completeness, we have included below the details of Hutter’s algorithm as they appear in [2].

The main idea of Hutter’s algorithm is to search the space of all proofs111Given a formal logic system $F$ with a set of axioms and inference rules. A proof in $F$ is a sequence of formulas, where each formula is either an axiom in $F$ or is something that can be inferred using $F$’s axioms and inference rules, and previoulsy inferred formulas. in some formal axiomatic system, and not the space of all programs. In particular, the algorithm searches for those proofs that can tell us which programs are provably equivalent to $p^{*}$ and have provably quickly computable time bounds. This is doable since the set of all proofs in any formal sysem is enumerable. Moreover, in [2], Hutter showed how to formalize the notions of provability, Turing Machines, and computation time. Let $U$ be a fixed reference Universal Turing machine. For finite binary strings $p$ and $t_{p}$, we assume that

• •

$U(p,x)$ computes the output of program $p$ on input $x$.

• •

$U(t_{p},x)$ returns a time bound on the runtime of program $p$ on input $x$.

Moreover, Hutter defined the following two terms:

• •

A term $u$ is defined such that the formula $[\forall y:u(p,y)=u(p^{*},y)]$ is true if and only if $U(p,x)=U(p^{*},x),\forall x$.

• •

A term $tm$ is defined such that the formula $[tm(p,x)=n]$ is true if and only if $U(p,x)$ takes $n$ steps; that is, if $time_{p}(x)=n$.

Then, we say that programs $p$ and $p^{*}$ are provably equivalent if the formula $[\forall y:u(p,y)=u(p^{*},y)]$ can be proved. The algorithm of Hutter is as follows. **Algorithm Hutter($x$)

**Let $L=\phi$, $t_{fast}=\infty$, and $p_{fast}=p^{*}$.

Run A,B, and C concurrently with 10%, 10%, and 80% of computational resources, respectively. **Algorithm A

**{ This algorithm identifies programs $p$ that are provably equivalent to $p^{*}$ and have provably computable time bounds $t_{p}$ }

for $i=1,2,3,\ldots$ do

pick the $i^{th}$ proof in the list of all proofs and

if the last formula in the proof is equal to $[\forall y,u(p^{*},x)=u(p,y)$ and

$u(t_{p},y)\geq tm(p,y)]$, for some pair of strings $(p,t_{p})$ then

Let $L=L\cup\{(p,t_{p})\}$.

End A **Algorithm B

**{ This algorithm finds the program with the shortest time bound in $L$.}

for all $(p,t_{p})$ in $L$ do

run $U$ on all $(t_{p},x)$ in parallel for all $t_{p}$ with relative computational

resources $2^{-l(p)-l(t_{p})}$.

if $U$ halts for some $t_{p}$ and $U(t_{p},x)<t_{fast}$ then

$t_{fast}=U(t_{p},x)$ and $p_{fast}=p$.

endfor all $(p,t_{p})$

End B **Algorithm C

**For $k=1,2,4,8,\ldots$ do

pick the currenlty fastest program $p_{fast}$ with time bound $t_{fast}$ .

run $U$ on $(p,x)$ for $k$ steps.

if $U$ halts in less than $k$ steps, then print result $U(p,x)$ and abort computation

of A, B, and C.

endfor $k$

End C In [2], it was shown that the overall runtime of Hutter’s algorithm is

[TABLE]

where $d_{p}=40.2^{l(p_{fast})+l(t_{p_{fast}})}$ and $c_{p}=40.2^{l(proof(p_{fast}))+1}.O(l(proof(p_{fast})^{2})$.

4 A Variation of Levin Search Inspired by the Fastest and Shortest Algorithm of Hutter

We rewrite Levin Search so that it searches, in parallel, the space of all proofs, and not the space of all programs. Let $p^{*}$ be a known fastest algorithm for some well-defined problem $\pi$. Then, our algorithm will construct a solution $p(x)$, for all $x$, that is provably equivalent to $p^{*}(x)$ in time $O(t_{p}(x))$, save for a factor of $O(l(proof(p))^{2})$, where $t_{p}(x)$ is a provable time bound on the runtime of the fastest program $p$ provably equivalent to $p^{*}$. We assume a formal axiomatic system $\mathbb{F}$ and the same terms $u$ and $tm$ defined in Hutter’s algorithm. Moreover, we assume that all pairs $(p,t_{p})$ are encoded using prefix-free codes, and hence, we have $\sum\limits_{(p,t_{p})}2^{-l(p)-l(t_{p})}\leq 1$, by Kraft inequality. The following is our Modified Levin Search algorithm: **Algorithm Modified Levin Search($x$)

$t=2$**; $t_{fast}=\infty$; $p_{fast}=\lambda$; { $p_{fast}$ is initialized to the empty string }

for all proofs $f\in F$ of length $l(f)=1,2,\ldots,\log t$ do

write down the first $t.2^{-l(f)}$ characters of proof $f$.

if the last formula in these characters is equal to $\forall y,u(p^{*},x)=u(p,y)$

and $u(t_{p},y)\geq tm(p,y)$, for some pair of strings $(p,t_{p})$ then

run $t_{p}(x)$ for at most $t.2^{-l(p)-l(t_{p})}$ steps.

if $t_{p}$ halts and $t_{p}(x)<t_{fast}$ then

$t_{fast}=t_{p}(x)$ and $p_{fast}=p$.

endfor

if $p_{fast}\not=\lambda$ then

run $p_{fast}$ for at most $t$ steps with time bound $t_{fast}$.

if $p_{fast}(x)$ halts, then return $p_{fast}$ for $p^{*}$ and halt.

endif

$t=2t.$

go to for all proofs $f\in F$

End of Modified Levin Search We note that during each time step $t$, Modified Levin Search runs at most a single provably fast program $p_{fast}$ that is provably equivalent to $p^{*}$. **Running time calculation: ** Following [2], let $n_{axioms}$ be the finite number of axioms in $\mathbb{F}$. Then, each proof $f\in\mathbb{F}$ is a sequence $F_{1}F_{2}\ldots F_{n}$, for some positive integer $n$, where each $F_{i}$ $(1\leq i\leq n)$ is either an axiom or a formula in $\mathbb{F}$. It takes $O(n_{axioms}.l(F_{i}))$ time to check if $F_{i}$ is an axiom, and $O(l(f))$ time to check if $F_{i}$ is a formula that can be inferred from other formulas $F_{j}\in f$, $j<i$. Thus, checking the validity of all formulas in $f$, and hence the validity of the proof $f$, takes $O(l(f)^{2})$ time. The “for” loop in Modified Levin Search takes time

[TABLE]

This is true because $\sum\limits_{(p,t_{p})}2^{-l(p)-l(t_{p})}\leq 1$, by Kraft inequality. we next have

[TABLE]

where $f^{\prime}$ is the proof that procuded the pair $(p,t_{p})$ with the shortest provable time bound $t_{p}(x)$, among all proofs of length $\leq\log t$. Thus, the runtime of Modified Levin Search is equal to

[TABLE]

where $l(f^{*})$ is the length of the binary encoding of the proof that produced the resultant pair $(p_{fast},t_{p_{fast}})$. Suppose that Modified Levin Search stops at time $t=T\leq t_{p_{fast}}(x)$.Then

[TABLE]

This shows that Modified Levin Search runs in time $O(t_{p_{fast}})$, save for a factor of $O(l(f^{*})^{2})$.

5 Concluding Remarks

We recall that Hutter’s algorithm was designed for well-defined problems. These are problems the solutions of which can be formally proved correct and have provably quickly computable time bounds. For programs provably correct (they halt and compute the correct answer), but for which no quickly computable time bounds exist (For example, the traveling salesman problem), Hutter [2] explained that an obvious time bound for a progam $p$ is its actual running time $time_{p}(.)$. By replacing $t_{p}(.)$ with $time_{p}(.)$ in the runtime of Hutter’s algorithm, and by noticing that, in this case, $time_{t_{p}}(x)$ becomes $time_{time_{p}}(x)\leq time_{p}(x)$, we have

[TABLE]

where $l(p)$ is the length of the binary encoding of $p$, and $l(f)$ is the length of the binary encoding of a proof $f$ that produces the program $p$ that is provably equivalent to the fastest known algorithm $p^{*}$ for the problem in hand. Thus, for such programs, Hutter’s algorithm is optimal, save for a huge constant multiplicative term and a huge constant additive term.

We next calculate the runtime of our algorithm in case we are dealing with programs that can be provably correct, but for which no quickly computable time bounds exist. Suppose that our algorithm stops at time $t=T$. Then, we have

[TABLE]

We also have $time_{p}(x)=T.2^{-l(p)}$. Thus,

[TABLE]

Thus, for such programs, the runtime of our algorithm also suffers from a huge constant multiplicative term, but with no additinoal huge additive terms.

We conclude this paper by recalling an approach, proposed by Solomonoff in 1985 [7], to manage the huge multiplicative constant in the big-oh notation of the running time of Levin Search. We do so because we think that the importance of that work of Solomonoff is not widely appreciated. Solomonoff argued that if machines are going to have a problem solving capability similar to that of humans, then machines cannot start from scratch everytime they attempt to solve a new problem. We, humans, rely on our previous knolwedge of solutions to other problems to figure out a solution for a new problem. The basic idea of Solomonoff is that we should be able to construct $p^{*}$ via a list of references to previously discovered solutions for other related problems. We can imagine writing the code for $p^{*}$ as $p_{1}p_{2}\ldots p_{n}$, where $p_{i}$ is a reference to a solution for problem $\pi_{i}$ stored on a separate work tape of a Kolmogorov-Uspensky Machine [4]. This way, the solution $p^{*}$ is a sequence of calls to other solutions stored on the work tapes. This way, the length of $p^{*}$ would be made significantly smaller than the sum of the lengths of $p_{i}$, and the saving in the length of $p^{*}$ would exponentially decrease the multiplicate constant in the big-oh notation of Levin Search.