Computational cost for determining an approximate global minimum using the selection and crossover algorithm

TL;DR

This paper analyzes how a selection and crossover algorithm can significantly reduce the computational cost of finding an approximate global minimum in complex, multi-minima cost functions, compared to random search.

Contribution

It demonstrates analytically and numerically that crossover-based genetic algorithms can lower the iteration count to find approximate minima, outperforming traditional random search methods.

Findings

Crossover reduces the required iterations from 2^N to λ^N, with λ<2.

Offspring distribution has lower mean cost than original samples.

Variance of offspring cost remains asymptotically unchanged, aiding search efficiency.

Abstract

This work examines the expected computational cost to determine an approximate global minimum of a class of cost functions characterized by the variance of coefficients. The cost function takes $N$-dimensional binary states as arguments and has many local minima. Iterations in the order of $2^N$ are required to determine an approximate global minimum using random search. This work analytically and numerically demonstrates that the selection and crossover algorithm with random initialization can reduce the required computational cost (i.e., number of iterations) for identifying an approximate global minimum to the order of $\lambda^N$ with $\lambda$ less than 2. The two best solutions, referred to as parents, are selected from a pool of randomly sampled states. Offspring generated by crossovers of the parents' states are distributed with a mean cost lower than that of the original distribution that generated the parents. It is revealed that in contrast to the mean, the variance of the cost of the offspring is asymptotically the same as that of the original distribution. Consequently, sampling from the offspring's distribution leads to a higher chance of determining an approximate global minimum than sampling from the original distribution, thereby accelerating the global search. This feature is distinct from the distribution obtained by a mixture of a large population of favorable states, which leads to a lower variance of offspring. These findings demonstrate the advantage of the crossover between two favorable states over a mixture of many favorable states for an efficient determination of an approximate global minimum.

Figures

A). It is analytically expressed that problems requiring the $2^{N}$-order time using a simple random search can be solved in the $\lambda^{N}$-order time, with $1<\lambda<2$, using the selection and crossover algorithm. This finding highlights the utility of selection and crossover for accelerating the global search.

A). The difference between a crossover of two states and a mixture of several states can be explained by the variances of their offspring (Fig. 3B). The variance of the distribution created by selection and crossover is large and converges to 1 as $K$ increases, indicating that sampling from the offspring’s distribution has sufficient variety. Whereas, the near-zero variance of the distribution created by the mean-field approximation indicates a near-zero variety of offspring. Therefore, only the selection and crossover algorithm can retain the variety of offspring and use it to enhance the se

Equations

$$F(x)=\sum _i{a}_i{x}_i+\sum _{i<j}{a}_{ij}{x}_i{x}_j+\sum _{i<j<k}{a}_{ijk}{x}_i{x}_j{x}_k+\cdots +\sum _i{a}_{1\dots N\backslash i}\prod _{j\ne i}{x}_j+a_{1\dots N}\prod _i{x}_i$$

$$=\sum _{1\le \alpha \le N}\left(\sum _{1\le i_1<\cdots <i_{\alpha }\le N}{a}_{i_1\dots i_{\alpha }}{x}_{i_1}\cdots x_{i_{\alpha }}\right).$$

$$a_{i_1\dots i_{\alpha }}\sim \mathcal{N}[{\mu }_{\alpha },{\sigma }_{\alpha }^2]$$

$$\mathrm{E}_{A}\Big{[}\mathrm{Var}_{X}[F(x)]\Big{]}=\sum_{1\leq\alpha\leq N}\binom{N}{\alpha}\sigma_{\alpha}^{2}=1$$

$$p_M(F_M)=M(1-\phi (F_M){)}^{M-1}p(F_M).$$

$$\frac{d{p}_M(F_M)}{d{F}_M}=-M(M-1)(1-\phi (F){)}^{M-2}p(F_M{)}^2+M(1-\phi (F){)}^{M-1}{\frac{dp(F)}{dF}|}_{F=F_M}=0.$$

$$Mp(F_M)\approx -\frac{F_M-\mu }{{\sigma }^2}.$$

$$M\approx \sqrt{\frac{2\pi (F_M-\mu {)}^2}{{\sigma }^2}}\exp \left(\frac{(F_M-\mu {)}^2}{2{\sigma }^2}\right).$$

$$\frac{(F_M-\mu {)}^2}{2{\sigma }^2}+\log (-F_M+\mu )=\frac{1}{2}\log \frac{M^2{\sigma }^2}{2\pi }.$$

$$F_M=\mu -\sigma \sqrt{\log \frac{M^2{\sigma }^2}{2\pi }-\log ({\sigma }^2\log M^2{\sigma }^2)}=\mu -\sigma \sqrt{\log \frac{M^2}{2\pi \log M^2{\sigma }^2}}.$$

$$p_M(F_M)=\mathcal{N}\left[\mu -\sigma \sqrt{\log \frac{M^2}{2\pi \log M^2{\sigma }^2}},{\sigma }^2{\left(\log \frac{M^2}{2\pi \log M^2{\sigma }^2}-1\right)}^{-1}\right]$$

$$\approx \mathcal{N}\left[\mu -\sigma \sqrt{2\log M},\frac{{\sigma }^2}{2\log M}\right].$$

$$p_{2^N}(F_{2^N})\approx \delta \left(F_{2^N}+\sqrt{2N\log 2}\right).$$

$$\mathrm{Prob}[z_i=1]={\rho }_i\equiv \frac{x_i^{(1)}+\cdots +x_i^{(n)}}{2n}+\frac{1}{2}.$$

$$\{\begin{array}{ll}{C}_0& \equiv \sum _{d+1\le i\le N}{a}_i{x}_i+\sum _{d+1\le i<j\le N}{a}_{ij}{x}_i{x}_j+\sum _{d+1\le i<j<k\le N}{a}_{ijk}{x}_i{x}_j{x}_k+\cdots \\ C_1(z)& \equiv \sum _{1\le i\le d}{a}_i{z}_i+\sum _{\begin{array}{c}1\le i\le d\\ d+1\le j\le N\end{array}}{a}_{ij}{z}_i{x}_j+\sum _{\begin{array}{c}1\le i\le d\\ d+1\le j<k\le N\end{array}}{a}_{ijk}{z}_i{x}_j{x}_k+\cdots \\ & ⋮\end{array}$$

$$C_{\alpha }(z)\equiv \sum _{1\le i_1<\cdots <i_{\alpha }\le d}{b}_{i_1\dots i_{\alpha }}{z}_{i_1}\dots z_{i_{\alpha }}$$

$$b_{i_1\dots i_{\alpha }}\equiv a_{i_1\dots i_{\alpha }}+\sum _{d+1\le j_1\le N}{a}_{i_1\dots i_{\alpha }{j}_1}{x}_{j_1}+\sum _{d+1\le j_1<j_2\le N}{a}_{i_1\dots i_{\alpha }{j}_1{j}_2}{x}_{j_1}{x}_{j_2}+\cdots .$$

$$\displaystyle\mathrm{E}_{A}\Big{[}\mathrm{Var}_{X^{*}}\left[b_{i_{1}\dots i_{\alpha}}\right]\Big{]}=\mathrm{E}_{A}[a_{i_{1}\dots i_{\alpha}}^{2}]+\!\!\sum_{d+1\leq j_{1}\leq N}\!\!\!\!\!\!\mathrm{E}_{A}[a_{i_{1}\dots i_{\alpha}j_{1}}^{2}]\mathrm{E}_{X^{*}}[x_{j_{1}}^{2}]+\cdots\hskip 31.29802pt$$

$$={\sigma }_{\alpha }^2+\left(\genfrac{}{}{0ex}{}{N-d}{1}\right){\sigma }_{\alpha +1}^2+\left(\genfrac{}{}{0ex}{}{N-d}{2}\right){\sigma }_{\alpha +2}^2+\cdots =\sum _{0\le \beta \le N-d}\left(\genfrac{}{}{0ex}{}{N-d}{\beta }\right){\sigma }_{\alpha +\beta }^2,$$

$$p(b_{i_1\dots i_{\alpha }})=\mathcal{N}\left[0,\sum _{0\le \beta \le N-d}\left(\genfrac{}{}{0ex}{}{N-d}{\beta }\right){\sigma }_{\alpha +\beta }^2\right].$$

$$\mathrm{E}_{A,X^{*}}\Big{[}\mathrm{E}_{Z}[C_{\alpha}(z)^{2}]\Big{]}=\binom{d}{\alpha}\sum_{0\leq\beta\leq N-d}\binom{N-d}{\beta}\sigma_{\alpha+\beta}^{2}.$$

$$\mathrm{E}_{A,X^{*}}\!\Big{[}\mathrm{Var}_{Z}[F(z)]\Big{]}=\mathrm{E}_{A,X^{*}}\!\Big{[}\mathrm{E}_{Z}[C_{1}(z)^{2}+\cdots+C_{d}(z)^{2}]\Big{]}=\sum_{1\leq\alpha\leq d}\!\!\binom{d}{\alpha}\!\!\sum_{0\leq\beta\leq N-d}\!\!\binom{N-d}{\beta}\sigma_{\alpha+\beta}^{2}.$$

$$\displaystyle\mathrm{E}_{A,X^{*}}\Big{[}\mathrm{Var}_{Z}[F(z)]\Big{]}=\sum_{0\leq\alpha\leq d}\binom{d}{\alpha}\sum_{0\leq\gamma\leq N}\binom{N-d}{\gamma-\alpha}\sigma_{\gamma}^{2}-\sum_{0\leq\beta\leq N-d}\binom{N-d}{\beta}\sigma_{\beta}^{2}$$

$$=\sum _{0\le \gamma \le N}\left(\genfrac{}{}{0ex}{}{N}{\gamma }\right){\sigma }_{\gamma }^2-\sum _{0\le \beta \le N-d}\left(\genfrac{}{}{0ex}{}{N-d}{\beta }\right){\sigma }_{\beta }^2=1-\sum _{0\le \beta \le N-d}\left(\genfrac{}{}{0ex}{}{N-d}{\beta }\right){\sigma }_{\beta }^2,$$

$$\{\begin{array}{l}\frac{F(x)+F(y)}{2}=C_0(x)+C_2(x)+C_4(x)+\cdots =-\sqrt{\log \frac{M^2}{2\pi \log M^2}}\\ \frac{F(x)-F(y)}{2}=C_1(x)+C_3(x)+C_5(x)+\cdots =0\end{array}$$

$$\frac{{\mathrm{E}}_A[C_0(x)]}{{\mathrm{Var}}_{A,X}[C_0(v)]}=\frac{{\mathrm{E}}_A[C_2(x)]}{{\mathrm{Var}}_{A,X}[C_2(v)]}=\frac{{\mathrm{E}}_A[C_4(x)]}{{\mathrm{Var}}_{A,X}[C_4(v)]}=\cdots ,$$

$${\mathrm{Var}}_{A,X}[C_0(v)+C_2(v)+\cdots ]=\sum _{\alpha =0,2,4,\dots }\sum _{1\le \beta \le N-d}\left(\genfrac{}{}{0ex}{}{N-d}{\beta }\right){\sigma }_{\alpha +\beta }^2\approx \frac{1}{2}\sum _{1\le \gamma \le N}\left(\genfrac{}{}{0ex}{}{N}{\gamma }\right){\sigma }_{\gamma }^2=\frac{1}{2}$$

$${\mathrm{E}}_A[C_0]=-2\left(\sum _{1\le \beta \le N/2}\left(\genfrac{}{}{0ex}{}{N/2}{\beta }\right){\sigma }_{\beta }^2\right)\sqrt{\log \frac{M^2}{2\pi \log M^2}}.$$

$$p(F)=\mathcal{N}\left[-2\eta \sqrt{\log \frac{M^2}{2\pi \log M^2}},1-\eta \right],$$

$$p_M(F_M)=\mathcal{N}\left[-\left(2\eta +\sqrt{1-\eta }\right)\sqrt{\log \frac{M^2}{2\pi \log M^2}},(1-\eta ){\left(\log \frac{M^2}{2\pi \log M^2}-1\right)}^{-1}\right].$$

$$M\approx 3\cdot {\lambda }^N,\mathrm{where}\lambda \equiv 2^{\frac{1}{(2\eta +\sqrt{1-\eta }{)}^2}}<2.$$

$$\frac{d^2{p}_M(F_M)}{d{F}_M^2}=M(M-1)(M-2)(1-\phi (F_M){)}^{M-3}p(F_M{)}^3$$

Full text

1 Introduction

Biological organisms occasionally outperform current machine learning techniques in terms of adaptive ability and robustness. This work examines the benefits of the search and optimization strategy used by organisms, selection and crossover [1, 2, 3, 4], toward the creation of a biologically-inspired optimization framework.

The intelligence of organisms refers to the optimization of search algorithms to identify optimal solutions. Organisms can recognize their surrounding environment by optimizing their internal representations about the dynamics and cases in the external world. In addition, they can optimize their behavior for adapting to the environment to increase the probability of survival and reproduction. These optimization problems are in many cases formulated as the minimization of a cost function. Theoretical neurobiology commonly uses cost functions to model neural activity [5, 6], various learning processes [7, 8, 9, 10, 11, 12], and evolution of nervous system [13]. A common optimization strategy under a cost function is gradient descent, a form of local search, that updates neural activities, synaptic strengths, and genes, to minimize the cost function. However, the gradient descent is not effective when the cost function has a huge number of local minima as a state is likely to be trapped in a bad local minimum.

Apart from local search algorithms, the selection and crossover algorithm, which is a nonlocal search method, has been popularized in the literature on genetic algorithm [1, 2, 3]. Crossovers can generate offspring that are sufficiently different from their parents, while inheriting some aspects of their favorable properties, when the parents’ states are sufficiently different from each other. This can provide a search strategy that is different from the local search algorithms, and may thus provide a higher chance of determining an approximate global minimum. Although the benefits of selection and crossover have been proposed as the building block hypothesis [14, 15], however, the conditions under which the selection and crossover algorithm accelerates the search remain unclear.

This work analytically and numerically explores the computational cost of the selection and crossover algorithm to determine an approximate global minimum, under the assumption that the output value of the cost function (i.e., its histogram) follows a Gaussian distribution (Fig. 1A). It is analytically expressed that problems requiring the $2^{N}$-order time using a simple random search can be solved in the $\lambda^{N}$-order time, with $1<\lambda<2$, using the selection and crossover algorithm. This finding highlights the utility of selection and crossover for accelerating the global search.

2 Methods

2.1 System and cost function

Here, we consider an optimization problem of $N$-dimensional binary states. A state is expressed using vector $x\equiv(x_{1},\dots,x_{N})\in X$ whose elements takes $-1$ or $1$, where $X\equiv\{-1,1\}^{N}$. A cost function of this system $F(x):X\mapsto\mathbb{R}$ maps $x$ to a real number that expresses the cost of position $x$ in $N$-dimensional space. The aim of this work is to identify $x$ that provides an approximate global minimum of $F(x)$. In general, $F(x)$ is parameterized by up to $2^{N}$ independent parameters; this is a general setup for combinational optimization problems [16]. Variables and parameter definitions are provided in Supplementary Table S1. In relation to neurobiology, $x$ can be assumed to represent neural activity, synaptic connections, or genes; thus, this optimization problem can become a model of perception, learning, or evolution, respectively.

We express $F(x)$ as the sum of products. Because $x_{i}$ takes only $-1$ or $1$, $F(x)$ is expanded as

[TABLE]

Here, coefficients $\{a_{i},a_{ij},a_{ijk},\dots,a_{1\dots N\backslash i},a_{1\dots N}\}$, or $a_{i_{1}\dots i_{\alpha}}$ in general using indices $i_{1},\dots,i_{\alpha}$, are assumed to independently follow Gaussian distributions:

[TABLE]

for $1\leq\alpha\leq N$. Without loss of generality, it is assumed that $\mu_{1}=\cdots=\mu_{N}=0$, so that $\mathrm{E}_{X}[F(x)]=0$, and $\mathrm{Var}_{X}[F(x)]=\mathrm{E}_{X}[F(x)^{2}]=1$, as the cost function can be rescaled prior to optimization. Here, $\mathrm{E}_{X}[\bullet(x)]$ refers to the expectation of $\bullet(x)$ when elements of $x$ are independently and randomly selected from the distribution over $X$. Let $x_{i}$ ($1\leq i\leq N$) take 1 with a probability of $1/2$. Immediately, as the expectation over various $a_{i_{1}\dots i_{\alpha}}\in A$, where $A$ indicates the distribution of coefficients, it holds that

[TABLE]

using the binomial coefficient $\binom{N}{\alpha}\equiv N!/(\alpha!(N-\alpha)!)$. In addition, it is assumed that $F(x)$ and $F(y)$ are nearly independent of each other when $x\neq y$. When $N$ is large, the distribution of $F(x)$ asymptotically approaches the unit Gaussian distribution, $p(F)=\mathcal{N}[0,1]$, according to the central limit theorem (Fig. 1A). Under these assumptions, in what follows, we will estimate the distribution of an approximate global minimum and the required computational cost to determine it.

2.2 Approximate global minimum and computational cost of random search

The optimization problem is simplified if only an approximate global minimum, not the exact global minimum, must be identified. The minimum among randomly selected $M$ samples from a set $\Omega\subseteq X$ can be estimated when their costs follow a Gaussian distribution. Thus, an approximate global minimum of our system can be defined as the minimum among randomly selected $2^{N}$ samples from $X$. Using the Laplace approximation, the probability distribution of the minimum of $M$-times random search is obtained as follows.

Let us independently and randomly select $M$ samples from set $\Omega$ and express them as $x^{(1)},\dots,x^{(M)}$. Their costs are supposed to follow an identical Gaussian distribution $p(F)=\mathcal{N}[\mu,\sigma^{2}]$ with mean $\mu$ and variance $\sigma^{2}$. The minimum among these $M$ samples is denoted by $F_{M}$. The cumulative distribution function of $F$ is defined as $\varphi(F)\equiv\int_{-\infty}^{F}p(F^{\prime})dF^{\prime}$. Moreover, the cumulative distribution function of $F_{M}$ is given by $\varphi_{M}(F_{M})\equiv\mathrm{Prob}[F_{M}\leq F(x^{(1)}),\dots,F(x^{(M)})]=1-\mathrm{Prob}[F_{M}>F(x^{(1)}),\dots,F(x^{(M)})]$. Since sampling is independent, it becomes $\varphi_{M}(F_{M})=1-\prod_{1\leq t\leq M}\mathrm{Prob}[F_{M}>F(x^{(t)})]=1-(1-\varphi(F_{M}))^{M}$. As the derivative of $\varphi_{M}(F_{M})$, the probability density of $F_{M}$ is given by

[TABLE]

The mode of $p_{M}(F_{M})$ satisfies

[TABLE]

When $F_{M}$ is sufficiently smaller than $\mu$, since $M\gg 1$ and $\varphi(F_{M})\ll 1$, (5) is approximated as

[TABLE]

Here, we used $dp(F)/dF=-(F-\mu)/\sigma^{2}p(F)$. Thus, we find the following theorem:

Theorem 1: required iterations for random search to obtain $F_{M}$.

When $F_{M}\ll\mu$, the required iterations $M$ to attain $F_{M}$, with a $1/2$ probability, by randomly selecting $F(x)$ from $p(F)=\mathcal{N}[\mu,\sigma^{2}]$ are

[TABLE]

Furthermore, taking the logarithm of (6) yields

[TABLE]

As the leading order, it approximately holds that $-F_{M}+\mu\approx\sqrt{\sigma^{2}\log{M^{2}\sigma^{2}}}$. By substituting it into $\log(-F_{M}+\mu)$ of (8), we obtain the mode of $p_{M}(F_{M})$ as

[TABLE]

Then, we approximate $p_{M}(F_{M})$ as a Gaussian distribution, $p_{M}(F_{M})=\mathcal{N}[\mu_{M},\sigma_{M}^{2}]$ (Laplace approximation). The mean, $\mu_{M}$, matches the mode of $p_{M}(F_{M})$. Moreover, from the second derivative of $p_{M}(F_{M})$, the variance of $p_{M}(F_{M})$ is solved as $\sigma_{M}^{2}=\sigma^{2}\left(\log(M^{2}/2\pi\log{M^{2}\sigma^{2}})-1\right)^{-1}$. See Supplementary Methods S1 for the derivation. Therefore, we find the following theorem:

Theorem 2: distribution of minimum among randomly selected M samples.

When $M$ samples are independently and randomly generated from an identical distribution $p(F)=\mathcal{N}[\mu,\sigma^{2}]$, the minimum among these samples, $F_{M}$, approximately follows

[TABLE]

From Theorem 2, we can obtain the probability of an approximate global minimum of the system. When $p(F)=\mathcal{N}[0,1]$ and $M=2^{N}$, the distribution of an approximate global minimum $F_{2^{N}}$ converges to a delta function

[TABLE]

When $F(x)$ involves only lower-order products, $F_{2^{N}}$ can be slightly close to zero due to a correlation between costs of neighboring states; however, the difference is expected to be small. In what follows, we will mathematically analyze the the computational cost to identify a state providing the above-mentioned approximate global minimum using the selection and crossover algorithm.

3 Theory on the computational cost of the selection and crossover algorithm

Here, it is demonstrated that the selection and crossover algorithm accelerates the global search. In this work, a crossover (or a mixture) of $n$ favorable states, $x^{(1)},\dots,x^{(n)}$, is expressed as an operation that randomly generates offspring $z=(z_{1},\dots,z_{N})\in Z$ by following the probability distribution

[TABLE]

The mean and variance of $z_{i}$ are $\mathrm{E}_{Z}[z_{i}]=2\rho_{i}-1$ and $\mathrm{Var}_{Z}[z_{i}]=1-\mathrm{E}_{Z}[z_{i}]^{2}=4\rho_{i}(1-\rho_{i})$; thus, we can see that $0\leq\mathrm{Var}_{Z}[z_{i}]\leq 1$ and the variance is maximized when and only when $\rho_{i}=1/2$. The case with $n=2$ provides the selection and crossover algorithm, while a large $n$ limit is known as the mean-field approximation, where elements of a state are assumed to be independent from each other. When $n=2$, it holds that $\rho_{i}=1/2$ when $x^{(1)}_{i}=-x^{(2)}_{i}$ and $\rho_{i}\in\{0,1\}$ otherwise. In contrast, when $n$ is larger than 2, $0<\rho_{i}<1$ and $\rho_{i}\neq 1/2$ hold for a generic cost function; thus, the variance is not maximized in this condition. Below, it is mathematically expressed that $n=2$ is the best to maximize the variance (i.e., variety) of offspring’s distribution; therefore, the selection and crossover algorithm can accelerate the global search.

3.1 Distribution of a crossover of two favorable states

Here, we consider the case with $n=2$. We randomly generate $M$ samples from $X$ and select the minimum, referred to as $x$; then, we randomly generate another $M$ samples and select the minimum, referred to as $y$ (Fig. 1B). The computational cost for this operation is $2M$. From Theorem 2, the results of $F(x)$ and $F(y)$ independently follow an identical Gaussian distribution (2.2).

We define a schema $x^{*}$ as a set of common elements between $x$ and $y$ (i.e., parents) and $Z$ as a set of $x$ that involves $x^{*}$. The expectation of $F(z)$ over $z\in Z$ (i.e., offspring) is likely to be lower than that over $x\in X$. We define the number of different elements by $d\equiv|x-y|^{2}/4$ using the Euclidean distance. Without loss of generality, we can suppose that $x_{i}$ and $y_{i}$ are different for $1\leq i\leq d$ (i.e., $x_{i}=-y_{i}$), while $x_{i}$ and $y_{i}$ are the same for $d+1\leq i\leq N$ (i.e., $x_{i}=y_{i}$). Thus, $z_{1},z_{2},\dots,z_{d}$ are randomly sampled from $\{-1,1\}^{d}$ while $(z_{d+1},\dots,z_{N})=(x_{d+1},\dots,x_{N})$ is fixed. This can be viewed as a problem with a $d$-dimensional state space.

It is possible to decompose $F(z)$ by the number of random variables $(z_{1},z_{2},\dots,z_{d})$ involved in a term. We define terms involving the $\alpha$-th order products of $z_{1},z_{2},\dots,z_{d}$ as follows:

[TABLE]

In general, these terms are expressed as

[TABLE]

for $0\leq\alpha\leq d$, where coefficients are given as

[TABLE]

The mean of $b_{i_{1}\dots i_{\alpha}}$ becomes $\mathrm{E}_{A,X^{*}}[b_{i_{1}\dots i_{\alpha}}]=0$ since $a_{i_{1}\dots i_{\alpha}},a_{i_{1}\dots i_{\alpha}j_{1}},\dots$ follow zero-mean distributions, where $X^{*}$ is a set of $x^{*}$, and the variance of $b_{i_{1}\dots i_{\alpha}}$ is calculated as

[TABLE]

since $a_{i_{1}\dots i_{\alpha}},a_{i_{1}\dots i_{\alpha}j_{1}},\dots$ are independent from each other. Thus, we find that $b_{i_{1}\dots i_{\alpha}}$ follows the following Gaussian distribution:

[TABLE]

Since $\mathrm{E}_{Z}[C_{\alpha}(z)]=0$ and $\mathrm{Var}_{Z}[C_{\alpha}(z)]=\mathrm{E}_{Z}[C_{\alpha}(z)^{2}]=\sum_{1\leq i_{1}<\cdots<i_{\alpha}\leq d}b_{i_{1}\dots i_{\alpha}}^{2}$ for $1\leq\alpha\leq d$, it holds that

[TABLE]

Thus, $F(z)$ is expressed as $F(z)=C_{0}+C_{1}(z)+C_{2}(z)+\cdots+C_{d}(z)=\sum_{0\leq\alpha\leq d}C_{\alpha}(z)$. The distribution of $F(z)$ over $Z$ has the mean $\mathrm{E}_{Z}[F(z)]=C_{0}$ and the variance $\mathrm{Var}_{Z}[F(z)]=\mathrm{E}_{Z}[F(z)^{2}]-C_{0}^{2}=\mathrm{E}_{Z}[(C_{1}(z)+\cdots+C_{d}(z))^{2}]$. If we assume that $C_{1}(z),\dots,C_{d}(z)$ are independent from each other, from (18), we obtain

[TABLE]

Using the Vandermonde’s identity, we find

[TABLE]

where $\gamma\equiv\alpha+\beta$. The last equality holds from (3).

Next, we estimate the expectation of $C_{0}$ over various $a_{i_{1}\dots i_{\alpha}}\in A$. From $F(x)=\sum_{0\leq\alpha\leq d}C_{\alpha}(x)$, $F(y)=\sum_{0\leq\alpha\leq d}C_{\alpha}(y)$, and $(y_{1},\dots,y_{d})=-(x_{1},\dots,x_{d})$, it holds that

[TABLE]

Here, one can view that, in both $F(x)$ and $F(y)$, $C_{0}(x),C_{2}(x),C_{4}(x),\dots$ have been optimized to minimize the sum of them. Whereas, $C_{1}(x),C_{3}(x),C_{5}(x),\dots$ do not contribute to the optimization since the sum of them is zero, which implies that they are negligible when considering the expectation of a cost over various $a_{i_{1}\dots i_{\alpha}}\in A$. Thus, we will drop them from evaluation. It is further assumed that the optimization using the random search is unbiased in the sense that the expectations of $-C_{0}(x),-C_{2}(x),-C_{4}(x),\dots$ over various $a_{i_{1}\dots i_{\alpha}}\in A$ are simply proportional to their magnitudes. Expressed differently, we assume

[TABLE]

where $x$ indicates the solution of the above-mentioned random search given a particular cost function, while $v\in X$ expresses a state randomly selected from $X$. This assumption can be viewed as a flat prior belief about cost functions from a Bayesian perspective.

Under these assumptions, it holds that $\mathrm{E}_{A}[C_{0}(x)]=\mathrm{E}_{A}[C_{0}(x)+C_{2}(x)+\cdots]\cdot\mathrm{Var}_{A,X}[C_{0}(v)]/\mathrm{Var}_{A,X}[C_{0}(v)+C_{2}(v)+\cdots]$. For a large $N$, it is approximated that

[TABLE]

from the Vandermonde’s identity and (3). Moreover, for a large $N$, $d$ converges to $d=N/2$. Therefore, the expectation of $C_{0}$ is estimated as

[TABLE]

Thus, under the Laplace assumption, we obtain the following theorem:

Theorem 3: distribution of crossovers.

Under the above-mentioned assumptions, $z\in Z$ provides $F(z)$ that follows

[TABLE]

where $\eta\equiv\sum_{1\leq\beta\leq N/2}\binom{N/2}{\beta}\sigma_{\beta}^{2}$. Therefore, when $M$ samples are independently and randomly generated from offspring’s distribution, from Theorem 2, the minimum among these samples, $F_{M}$, approximately follows

[TABLE]

Hence, by solving $-(2\eta+\sqrt{1-\eta})\sqrt{2\log{M}}=-\sqrt{2N\log{2}}$, we find that

Theorem 4: computational cost for the selection and crossover algorithm.

The computational cost (total iterations) $M$ to attain an approximate global minimum using the selection and crossover algorithm are

[TABLE]

In summary, we have derived the distribution of offspring generated by a crossover of two favorable states, which is characterized by negative mean and asymptotically unit variance. Consequently, the selection and crossover algorithm reduces the computational cost for the global search to $\mathcal{O}(\lambda^{N})$.

3.2 Distribution of a mixture of many favorable states

For comparison, when $n$ is large, the mean and variance of offspring’s distribution are calculated as follows. Because $0<\rho_{i}<1$ for a generic cost function, from (2.1), we obtain the mean $\mathrm{E}_{Q}[F(z)]=R\equiv\sum_{1\leq\alpha\leq N}\left(\sum_{1\leq i_{1}<\cdots<i_{\alpha}\leq N}a_{i_{1}\dots i_{\alpha}}\rho_{i_{1}}\cdots\rho_{i_{\alpha}}\right)\neq 0$, where the probability of $z$, $Q\equiv Q(z)=\prod_{1\leq i\leq N}\mathrm{Prob}(z_{i})$, is characterized by $\rho_{1},\dots,\rho_{N}$. Moreover, we obtain the variance $\mathrm{Var}_{Q}[F(z)]=1-\mathrm{E}_{Q}[F(z)]^{2}=1-R^{2}<1$. From (9), the expectation of the minimum among randomly selected $M$ samples from the distribution is $F_{M}=R-\sqrt{(1-R^{2})\cdot 2\log{M}}\geq-\sqrt{1+2\log{M}}$, where $F_{M}=-\sqrt{1+2\log{M}}$ holds when and only when $R=-1/\sqrt{1+2\log{M}}$. Therefore, a mixture of many favorable states is, at most, the same performance as the random search for a large $M$, and thus ineffective for the global search.

4 Simulation and results

The accuracy of the above-mentioned theoretical predictions is examined by numerical simulations (codes are appended as Supplementary Source Codes). Here, we consider a class of the $K$-th order cost functions comprising up to the $K$-th order products of elements of $x$. Figure 2 illustrates the relationships between the dimensions of state space $N$ and the value of the global minimum, the solutions of random search and selection and crossover algorithm, and the mean of offspring’s distribution obtained by a crossover of two favorable states, when the second-, third- and fourth-order cost function is considered (A-C). First, the global minimum becomes large negative in the order of $\sqrt{N}$ as predicted by (11), although the global minimum for $K=2$ is slightly larger than the prediction due to the considerable correlation between neighboring states. The random search provides a solution with a cost determined only by the number of iterations ($M$), as shown in (2.2), regardless the order $K$ and dimensions $N$, unless it finds the global minimum.

Crucially, the selection and crossover algorithm finds a solution with a cost $F$ lower than that of the random search, while their computational costs are the same. The outcome of using the selection and crossover algorithm is characterized by the gain $g\equiv(2\eta+\sqrt{1-\eta})^{2}>1$, where the search efficacy of the selection and crossover algorithm with the computational cost $M$ is equivalent, in the order, to that of iterating random searches $M^{g}$ times, providing a higher chance to find the global minimum. The gain becomes close to 2 when $K=2$, while it converges to 1 when $K$ is large due to an increasing complexity of a cost function. The mean of the offspring’s distribution is proportional to $-\eta$, where $\eta$ converges to $2^{-K}$ for the $K$-th order cost functions when $N$ is large. These properties are as predicted by Theorem 3: see (25) and (26).

In contrast, when we generate offspring following a mixture of several favorable states, as an mean-field approximation of their probability distribution, the best solution among the offspring’s distribution is much worse (i.e., higher) than that obtained by a crossover of two favorable states, while both methods iterate the selection of parents and offspring with the same number $M$ (Fig. 3A). The difference between a crossover of two states and a mixture of several states can be explained by the variances of their offspring (Fig. 3B). The variance of the distribution created by selection and crossover is large and converges to 1 as $K$ increases, indicating that sampling from the offspring’s distribution has sufficient variety. Whereas, the near-zero variance of the distribution created by the mean-field approximation indicates a near-zero variety of offspring. Therefore, only the selection and crossover algorithm can retain the variety of offspring and use it to enhance the search efficacy.

5 Discussion and conclusion

An optimization problem comprising a cost function and discrete states can be solved in principle; however, the computational cost may be large, in the worst case increasing to the $2^{N}$ order of the dimensions of states $N$. Consequently, in practice, the optimization problem cannot be solved. This work analytically and numerically demonstrates the computational cost of the selection and crossover algorithm for determining an approximate global minimum of a class of cost functions characterized by the variance of coefficients. The use of the gradient descent search should accelerate the global search. In future work, the computational cost for an approach combining the gradient descent search and the selection and crossover algorithm will be addressed.

While the selection and crossover algorithm has been popularized as a genetic algorithm [1, 2, 3], it should be emphasized that it can be a computational model for insightful search in the brain [17, 18]. Creating a new state with a crossover of two existing states may be more relevant than randomly coming up with a new state, as a function of insight, in the sense that only the crossover can enhance the search efficacy. The present work is potentially interesting because optimizations in neurobiology are in most cases modeled as local searches; thus, the selection and crossover algorithm may provide an explanation about optimizations beyond local searches which the brain may perform.

In summary, it is shown that when a random search needs iterations in the order of $2^{N}$ to determine an approximate global minimum, the selection and crossover algorithm can reduce the computational cost to the order of $\lambda^{N}$ with $\lambda$ less than 2. Unlike a mixture of several favorable states, a crossover of two favorable states can provide a distribution of offspring with negative mean and variance that converges to 1. This highlights an advantage of a crossover of two favorable states, as opposed to a mixture of many favorable states, in efficiently determining an approximate global minimum.

Acknowledgments

This work was supported by RIKEN Center for Brain Science.

Supplementary Information

Computational cost for determining an approximate global minimum using the selection and crossover algorithm

Supplementary Tables

Supplementary Methods

S1. Derivation of variance of $p_{M}(F_{M})$

Since the first derivative of $p_{M}(F_{M})$ is expressed as (4), the second derivative is obtained as

[TABLE]

Since $M\gg 1$ and $\varphi(F_{M})\ll 1$, it holds that

[TABLE]

Here, in the second line, we used (6). Moreover, when $p_{M}(F_{M})=\mathcal{N}[\mu_{M},\sigma_{M}^{2}]$, the second derivative is given by $d^{2}p_{M}(F_{M})/dF_{M}^{2}=-1/\sigma_{M}^{2}\cdot p_{M}(F_{M})+(F_{M}-\mu_{M})^{2}/\sigma_{M}^{4}\cdot p_{M}(F_{M})$. Thus, when $F_{M}=\mu_{M}$, from (9), it is obtained that

[TABLE]