Network Horizon Dynamics I: Qualitative Aspects

TL;DR

This paper explores the qualitative connectivity properties and phase transitions in mostly acyclic directed networks, with applications spanning machine learning, cosmology, and network design, highlighting novel horizon growth phenomena.

Contribution

It introduces a qualitative analysis of phase transitions in directed networks, proposing hybrid designs and a small-world approach to horizon problems in cosmology.

Findings

Identification of three types of phase transitions affecting horizon sizes

Proposal of hybrid local/random network designs with connectivity benefits

Introduction of a small-world horizon approach in early universe cosmology

Abstract

Mostly acyclic directed networks, treated mathematically as directed graphs, arise in machine learning, biology, social science, physics, and other applications. Newman [1] has noted the mathematical challenges of such networks. In this series of papers, we study their connectivity properties, focusing on three types of phase transitions that affect horizon sizes for typical nodes. The first two types involve the familiar emergence of giant components as average local connectivity increases, while the third type involves small-world horizon growth at variable distance from a typical node. In this first paper, we focus on qualitative behavior, simulations, and applications, leaving formal considerations for subsequent papers. We explain how such phase transitions distinguish deep neural networks from shallow machine learning architectures, and propose hybrid local/random network designs with surprising connectivity advantages. We also propose a small-world approach to the horizon problem in the cosmology of the early universe as a novel alternative to the inflationary hypothesis of Guth and Linde.

Equations

$$P_N^3(K)=\frac{1}{\left(\genfrac{}{}{0ex}{}{2N^2}{K}\right)}\sum _{n=1}^N(-1{)}^{n+1}(\genfrac{}{}{0ex}{}{N}{n}\left)\right(\genfrac{}{}{0ex}{}{2N^2-2n}{K-2n})\text{ and }{\mathcal{P}}_N^3(p)=1-{\left(1-p^2\right)}^N.$$

$$|{\sigma }_m(x)|\approx \text{\# chains from x to generation m}\approx {\left(\frac{k}{N}\right)}^m\approx {\left(pN\right)}^m.$$

$$P_N^M(K)\approx 1-\exp \left(-\frac{K^{M-1}}{(M-1{)}^{M-1}{N}^M}\right)\text{ and }{\mathcal{P}}_N^M(p)\approx 1-\exp \left(-p^{M-1}{N}^{M-2}\right).$$

$$h(n):=N-N{\left(1-\frac{k}{N^2}\right)}^n.$$

$$H(m+1)=N\left(1-q^{H(m)}\right),$$

$$|{\sigma }_m(x)|\approx \gamma N\left(1-\exp \left(-\frac{k^{m-1}}{N^m}\right)\right),$$

$$1-{\left(1-\frac{k}{N^2}\right)}^n=\sum _{j=1}^n(-1{)}^{j+1}(\genfrac{}{}{0ex}{}{n}{j}){\left(\frac{k}{N^2}\right)}^j.$$

$$\frac{1}{\left(\genfrac{}{}{0ex}{}{N^2}{k}\right)}\sum _{j=1}^n(-1{)}^{j+1}(\genfrac{}{}{0ex}{}{n}{j}\left)\right(\genfrac{}{}{0ex}{}{N^2-j}{k-j})=\sum _{j=1}^n(-1{)}^{j+1}\left(\genfrac{}{}{0ex}{}{n}{j}\right)\prod _{\ell =0}^{j-1}\frac{k-\ell }{N^2-\ell }.$$

$$P_{N}^{M}(K)\approx 1-\left(1-\frac{k}{N^{2}}\right)^{N\Big{(}1-\exp\big{(}-\frac{k^{M-2}}{N^{M-1}}\big{)}\Big{)}}\approx 1-\left(1-\frac{k}{N^{2}}\right)^{\big{(}\frac{k}{N}\big{)}^{M-2}},$$

$$P(x_0\to x_{M-1})\approx 1-\left(1-\frac{K}{\left(\genfrac{}{}{0ex}{}{M}{2}\right)N^2}\right)\exp \left(-\sum _{m=0}^{M-3}\frac{\left(\genfrac{}{}{0ex}{}{M-2}{m}\right)K^{M-m-1}}{{\left(\genfrac{}{}{0ex}{}{M}{2}\right)}^{M-m-1}{N}^{M-m}}\right),$$

$$P(x_0\to x_{M-1})\approx 1-\exp \left(-\sum _{m=0}^L\frac{\left(\genfrac{}{}{0ex}{}{M-2}{m}\right)K^{M-m-1}}{{\left(\genfrac{}{}{0ex}{}{M}{2}\right)}^{M-m-1}{N}^{M-m}}\right).$$

$$|{\sigma }_2(x_0)|\approx N_2-N_2{\left(1-\frac{k_{12}}{N_1{N}_2}\right)}^{\frac{k_{01}}{N_0}}.$$

$$P(x_0\to x_2)\approx 1-\exp \left(-N_1{p}^2\right)\approx 1-\exp \left(-\frac{K^2}{N_1(N_0+N_2{)}^2}\right).$$

$$|\sigma_{M-1}(x_{0})|\approx N_{M-1}-N_{M-1}\big{(}1-p\big{)}^{N_{M-2}\Big{(}1-\exp\big{(}p^{M-2}\prod_{j=1}^{M-3}N_{j}\big{)}\Big{)}}.$$

$$P(x_0\to x_{M-1})\approx 1-\exp \left(-p^{M-2}\prod _{j=1}^{M-2}{N}_j\right)$$

$$\approx 1-\exp\left(-K^{M-1}\prod_{j=1}^{M-2}N_{j}\Big{/}\Big{(}\sum_{j=0}^{M-2}N_{j}N_{j+1}\Big{)}^{M-1}\right).$$

Full text

1 Introduction

Mostly acyclic directed networks are networks in which information, energy, or some other quantity flows in essentially one direction, though with possible small loops back. These are necessary to model features such as local feedback in a deep neural network [3, 4, 5, 6], eddies in a fluid flow [7, 8], papers or patents that cite themselves or each other [9, 10, 11, 12], pairs of species that prey upon each other [13, 14, 15], and closed causal curves in spacetime geometry near black holes [16, 17]. Figure 1 illustrates a mostly acyclic directed network. Newman [2] remarks about such networks that “not that many of their properties are known–calculations for these networks appear to be harder in many ways than for other types of random graphs.” Here, we study connectivity properties of such networks, combining results about specific simple networks with generalization strategies involving graph construction and decomposition methods. For brevity, we work informally, leaving detailed arguments to subsequent papers. After outlining results for strictly graded networks in Section 1, we discuss more general networks in Section 2, and applications in Section 3.

1.1 Background and Motivation

Our goal is to understand how connection probabilities for pairs of network nodes change as edges are added or deleted, usually randomly, or as the distance between nodes increases in some frame of reference. This allows us to model horizons, which measure accessibility from a given node. Connection probabilities are prominent in random graph theory [18, 19, 20, 21], network theory [2, 22, 23, 24], and percolation theory [25, 26, 27, 28, 29]. They depend on the proportion and distribution of permissible edges actually included in a network, where permissibility depends on the choice of model. For example, in early percolation theory, nodes often represented lattice points in a low-dimensional geometry used to model a physical system such as a porous medium. Edges between distant nodes were forbidden, producing a local structure. By contrast, information-theoretic networks such as social networks and artificial neural networks can be highly nonlocal.

Connection probabilities are subject to at least three different types of phase transitions. The first two types result from randomly increasing average local connectivity, while the third type involves small-world growth of a sequence of sub-networks. First is a topological phase transition, in which a giant connected component abruptly emerges as the proportion of permissible edges increases. In a directed network, nodes in the giant component are seldom accessible from each other immediately after this transition, since the paths connecting them are seldom directed paths. The giant component is therefore called weakly connected. Second, for a directed network, an accessibility phase transition occurs as the proportion of permissible edges continues to increase, with a large proportion of nodes abruptly becoming accessible from a given node along directed paths. This transition typically produces a giant in-component, from which a large proportion of nodes are accessible, a giant out-component, which is accessible from a large proportion of nodes, and their intersection, the giant strongly-connected component, which contains directed paths in both directions between any pair of its nodes. These components may be represented schematically by the “bow-tie” diagram of Broder et al. [30]. Third, a sequence of sub-networks of a network may exhibit a small-world phase transition, in which a large proportion of the entire network is included in every member of the sequence beyond a certain critical index range. The type of sub-network of principal interest in this paper consists of all nodes connected to a given node $x$ via directed paths of length $m$, which we call the degree-$m$ horizon of $x$, and denote by $\sigma_{m}(x)$. For example, Figure 1 shows a degree-$2$ horizon. Negative $m$ index past horizons, defined by following directed paths backwards from $x$, while positive $m$ index future horizons.

Despite much progress in the theory of phase transitions for directed/oriented percolation in lattices, less emphasis has been placed on mostly acyclic directed networks in general. Instead, the literature focuses on highly cyclic networks, such as the human brain, social networks, and the Internet. Technically, a mostly acyclic directed network is a network well-approximated by its simple acyclic reduction, given by reducing strongly-connected components to nodes. A typical example of such a network is an interconnected family of food chains in an ecological community, called a food web. Typical pairs of species either exhibit unambiguous predator-prey relationships or coexist amicably, but a few pairs such as alligators and pythons [15] prey upon each other. Locally recurrent neural networks also offer good examples. More profound examples are discrete spacetime models, which may exhibit cycles near black holes or at the Planck scale, but which obey an irreversible arrow of time in ordinary regimes. While such networks cannot develop giant strongly-connected components, they still undergo accessibility phase transitions, in which a large proportion of the “future” becomes accessible, and a large proportion of the “past” becomes visible, from a typical node. They also exhibit small-world phase transitions, unless limited by ambient low-dimensional geometry, as in early percolation theory.

Among many applications of these ideas, we discuss two of particular interest. First, deep neural networks (DNNs) have enjoyed a recent resurgence in popularity, after being temporarily eclipsed by alternative machine learning architectures [6, 31, 32]. Such networks are characterized by the use of deep layers of artificial neurons that progressively filter data flow between input and output layers. Their functional connectivity properties are determined by weights assigned to their edges. Networks are trained to produce desired outputs by adjusting their weights via numerical methods such as gradient descent. Training induces functional suppression of insignificant or redundant edges, which may be supplemented by explicit pruning [33, 39]. Such processes are analogous to biological synaptic pruning, an important aspect of maturation [34, 35]. Phase transition theory facilitates a balance between minimizing redundancy and preserving connectivity in network design. Shallow networks lacking deep layers can only undergo topological phase transitions, but deep networks can undergo all three types. Knowledge of this behavior may enable significant design improvements. In particular, hybrid local/random networks constructed by augmenting convolutional neural networks [36, 37, 38] with sparse random structure exhibit superior connectivity properties at reduced computational cost. Novel pseudorandom designs have already eclipsed standard architectures in accuracy and efficiency [39, 40, 41]. Such methods may allow construction of networks capable of next-generation tasks such as recognition of individuals among a large population, while democratizing access to state-of-the art technology.

Second, something very like a small-world phase transition seems to characterize the connectivity properties of the early universe, to judge by standard astronomical data. Evidence for this strange horizon problem was first noted by Rindler [42] and later strengthened by the discovery and analysis of the cosmic microwave background [43, 44] around the same time that Erdös and Rényi popularized random graph theory [45]. The resulting explanatory issues later motivated the inflationary hypothesis of Guth [46] and Linde [47]. While the data itself exhibits tell-tale signs of small-world behavior familiar in modern network theory, initial analysis of the problem preceded both the development of discrete spacetime models involving networks [48, 49, 50, 51, 52], and the general theory of phase transitions for directed networks [19, 22]. Perhaps for this reason, small-world phase transitions have not been seriously investigated as a possible solution to the horizon problem until quite recently [53, 54, 55], although other types of phase transitions are routinely studied in early cosmology [56, 57, 58]. General mechanisms such as Feynman’s path summation approach to quantum theory [59] and the phenomenon of preferential attachment in network growth processes [2] tend to strengthen the case for a small-world explanation of the horizon problem. This topic is of significant present interest, with recent data disfavoring simple inflationary models [60, 61, 62, 63].

1.2 Strictly Graded Networks and their Graphs

We begin by introducing strictly graded networks, whose edges connect nodes of neighboring grade levels, called generations. From an abstract mathematical viewpoint, such networks are strictly graded directed graphs. Figure 2 illustrates a $3$-generation graph with $5$ nodes per generation. Edge directions are “up the page,” in analogy to the arrow of time in Minkowski spacetime diagrams.

We denote a typical node of generation $m$ by $x_{m}$. Permissible edges are of the form $x_{m}x_{m+1}$, where $x_{m}$ is the initial node and $x_{m+1}$ is the terminal node. We focus first on two closely-related random graph models $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$, each with $M$ generations of size $N$.

Definition 1**.**

$M$-generation graphs* $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$ are strictly graded directed graphs with $M$ generations of $N$ nodes each. $G_{N}^{M}(K)$ has $K$ randomly-chosen edges, and $\mathscr{G}_{N}^{M}(p)$ includes each edge with probability $p$.*

It is convenient to number the initial and final generations of an $M$-generation graph as [math] and $M-1$ rather than $1$ and $M$. The relationship between $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$ is that $K$ is the expected number of edges when $p=\frac{K}{(M-1)N^{2}}$. $G_{N}^{M}(K)$ is convenient for describing graph construction processes, while $\mathscr{G}_{N}^{M}(p)$ is convenient for certain counting arguments. Both may be viewed as configuration spaces equipped with appropriate probability distributions, but we work mostly with individual graphs. We later generalize to allow generations of different sizes, generation-skipping edges, multiple edges between a given pair of nodes, and small cycles. This provides the tools necessary to analyze connectivity properties of arbitrary mostly acyclic networks. However, our main interest is in generic properties of specific classes of large networks arising in applications.

1.3 Connection Probabilities and Horizons

Applications such as artificial neural networks motivate study of information flow from “beginning to end” in a mostly acyclic directed network. For $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$, this involves the probabilities $P_{N}^{M}(K)$ and $\mathscr{P}_{N}^{M}(p)$ of connection between typical initial and final-generation nodes $x_{0}$ and $x_{M-1}$. A chain $c=x_{0}x_{1}...x_{M-1}$ connecting $x_{0}$ and $x_{M-1}$ is a sequence of edges $x_{0}x_{1},x_{1}x_{2},...,x_{M-2}x_{M-1}$, where the terminal node of each edge is the initial node of the next. Chains are special directed paths that include each of their edges exactly once. The length of $c$ is its number of edges $M-1$, and it is called an $(M-1)$-chain. A $2$-chain $x_{0}x_{1}x_{2}$ is illustrated in the right-hand diagram in Figure 2. Connection probabilities for $M=1$ are trivial, since there are no edges. For $M=2$, the bipartite case, $P_{N}^{2}(K)=\frac{K}{N^{2}}$, and $\mathscr{P}_{N}^{2}(p)=p$. For $M>2$, connection probabilities quickly become more complicated, with $P_{N}^{M}(K)$ generally worse than $\mathscr{P}_{N}^{M}(p)$. For example,

[TABLE]

For $p=\frac{K}{(M-1)N^{2}}$, the probabilities $P_{N}^{M}(K)$ and $\mathscr{P}_{N}^{M}(p)$ (rescaled) nearly coincide for large $N$. The left-hand diagram in Figure 3 shows plots of $P_{N}^{3}(K)$ (blue) versus $\mathscr{P}_{N}^{3}(\frac{K}{2N^{2}})$ (red, abscissa units $\frac{1}{2N^{2}}$) for $N=3,4,$ and $5$. The dashed lines indicate the maximum number of permissible edges in each case. As $N$ increases, the growth in the connection probabilities from near [math] to near $1$ occurs over a progressively shorter proportion of the interval involved, due to an accessibility phase transition elaborated below. The transition becomes arbitrarily sharp for large $N$. The right-hand diagram shows the nearly step-function behavior of $\mathscr{P}_{N}^{3}(\frac{K}{2N^{2}})$ for $N=10^{4}$.

For any directed graph, the degree-$m$ horizon $\sigma_{m}(x)$ of a node $x$ is the “cross-section” of nodes accessible from $x$ via chains of length $m$. In Figure 2, $\sigma_{1}(x_{0})=\{x_{1}\}$, and $\sigma_{2}(x_{0})=\{x_{2},x_{2}^{\prime}\}$. In relativistic physics, discussed in Section 3.2,“horizons” in spacetime are defined to be boundaries rather than cross sections, but either notion suffices for studying connectivity. Our definition is motivated by the fact that it is easier to generalize cross sections to a non-geometric setting than boundaries. We now sketch a heuristic argument that the horizon size $|\sigma_{m}(x)|$ for a node $x$ in $G_{N}^{M}(K)$ or $\mathscr{G}_{N}^{M}(p)$ behaves qualitatively like a logistic population model: it scales exponentially with increasing generation number unless it is limited by the generational size $N$, which plays the role of a “carrying capacity.” This is a familiar type of argument for random and pseudorandom graphs [2, 21]. For simplicity, we assume that $x$ belongs to generation [math]. Typical neighboring generations are connected by roughly $k=\frac{K}{M-1}\approx pN^{2}$ edges each, so roughly $\frac{k}{N}\approx pN$ edges connect $x$ to generation $1$. Working inductively, the number of chains between $x$ and generation $m$ is roughly $\frac{k}{N}\approx pN$ times the number of chains between $x$ and generation $m-1$. For small $k$ or $p$, these chains typically terminate at different nodes, yielding the estimates

[TABLE]

Initial exponential growth, rather than decay, is expected if $k>N$ or $p>\frac{1}{N}$. Significant horizon size is expected whenever (1.2) is of order $N$, which occurs whenever $k\approx N^{\frac{m+1}{m}}$ or $p\approx N^{-\frac{m-1}{m}}$. Once growth is halted by the generation size $N$, the average size of $\sigma_{m}(x)$ gradually decays unless $k\approx N^{2}$ or $p\approx 1$. This is not because typical horizon sizes gradually decay, but because the naïve constant asymptotic expected size is subject to random fluctuations, and zero is an attractor in the sense that if $|\sigma_{n}(x)|=0$ for some $n$, then $|\sigma_{m}(x)|=0$ for all $m>n$. The half-life of this decay process can be quite long, and it is therefore irrelevant in many applications. By contrast, the initial exponential behavior predicted by (1.2) is pivotal. It differs markedly from what occurs in geometry, where horizons grow roughly like a fixed power, namely, the dimension of the manifold. It is also unlike percolation in a lattice, since lattices and manifolds share similar notions of locality. Random graded graphs have no such notion; all pairs of nodes in neighboring generations are equally likely to be connected. Hence, change from random to geometric structure induces change from exponential to power-law scaling of horizons. This principle has significant consequences in applications such as machine learning and cosmology.

1.4 Three Phase Transitions

We now revisit the three types of phase transitions mentioned in Section 1.1, and discuss how these transitions affect connection probabilities and horizon sizes, focusing at first on $M$-generation graphs $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$.

1. Topological phase transition. If we randomly increase the proportion of permissible edges in an $M$-generation graph, a weakly connected giant component abruptly emerges and quickly grows to fill most of the graph. This is the topological phase transition. The green curves in Figure 4 show the growth of the giant component as $K$ increases for $3$-generation graphs $G_{10}^{3}(K)$ and $G_{100}^{3}(K)$. The connection probabilities $P_{N}^{M}(K)$ and $\mathscr{P}_{N}^{M}(p)$ remain small immediately after the topological phase transition, since typical connections in the giant component during its initial emergence involve undirected sequences of edges rather than chains. Since we are principally interested in connection probabilities and horizon sizes, we do not devote much attention to the topological phase transition.

2. Accessibility phase transition. If $M>2$, then the graph $G_{N}^{M}(K)$ or $\mathscr{G}_{N}^{M}(p)$ has “deep” or “hidden” generations. In this case, further increasing the proportion of permissible edges after the topological phase transition induces a second phase transition, during which the connection probability $P_{N}^{M}(K)$ or $\mathscr{P}_{N}^{M}(p)$ abruptly increases from near [math] to near $1$. This is the accessibility phase transition. Equivalently, the final horizon of a typical zeroth generation node abruptly expands to fill most of the final generation. Powerful generating function methods [19] exist for predicting accessibility phase transitions in random directed graphs obeying a specified degree distribution, but mostly acyclic networks such as $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$ are not easily analyzed via such methods because their degree distributions fail to characterize their prevailing unidirectional structure. Heuristic arguments explained in Section 2 suggest that accessibility phase transitions for such graphs occur near the critical values $K=(M-1)N^{\frac{M}{M-1}}$ and $p=N^{-\frac{M-2}{M-1}}$, respectively, and that the connection probabilities are approximately

[TABLE]

Proof in the case $M=3$ will appear in [10]. Informally, (1.3) says that for $M>2$, a very small proportion of permissible edges suffices to connect typical initial and final-generation nodes in a large $M$-generation graph.

An advantage of $G_{N}^{M}(K)$ over $\mathscr{G}_{N}^{M}(p)$ is that it may be regarded as the $K$th stage of a random edge-addition process, in which permissible edges are added sequentially to a preexisting node set. A dual process of edge removal is sometimes useful as well, for example, in training and/or pruning of artificial neural networks. $P_{N}^{M}(K)$ may be interpreted as the expected behavior of a counting function $F_{N}^{M}(K)$, which records the proportion of connections between initial and final-generation nodes in $G_{N}^{M}(K)$ as $K$ varies. The estimate (1.3) predicts a rapid increase in $F_{N}^{M}(K)$ with increasing $K$ near the critical value. Figure 4 shows examples of $F_{N}^{3}(K)$ (blue) for $N=10$ and $N=100$. Also shown are $P_{N}^{3}(K)$ (black), via (1.1), the estimate (1.3) (red), and the relative size of the weakly-connected giant component (green) from the earlier topological phase transition. For $N=100$, the blue, black, and red curves nearly coincide. The vertical lines show the center and radius of the phase transition, and the horizontal line shows the approximation $P_{N}(2N^{\frac{2}{3}})\approx\frac{e-1}{e}$ at the center, to be derived in [10].

In a more general setting, an accessibility phase transition may occur due to a network construction process that adds random edges to a preexisting edge structure, typically uniform, or that randomly changes a proportion of edges from a uniform to random arrangement. For example, rather than $G_{N}^{M}(K)$ or $\mathscr{G}_{N}^{M}(p)$, one may begin with a local lattice-like structure in which each node is connected to a small local kernel of nodes in the previous generation. This type of architecture characterizes convolutional neural networks, discussed further in Section 3.1. In such a network, each initial-generation node communicates with a fixed proportion of final-generation nodes, determined by a power law derived from the size and shape of the kernel. This proportion may be quite small, so such networks are often supplemented by tacking on computationally-expensive dense layers that include every permissible edge. However, adding a few random edges instead, or changing a few edges to a random arrangement, can induce an accessibility phase transition, leading to a highly connected hybrid local/random network.

3. Small-world phase transition. Following the accessibility phase transition, it becomes possible for the degree-$m$ horizon $\sigma_{m}(x)$ of a node $x$ in $G_{N}^{M}(K)$ or $\mathscr{G}_{N}^{M}(p)$ to approach order $N$ in size, i.e., to fill a significant proportion of its generation. Such a generation is then a small world from the perspective of $x$. A useful physics-motivated analogy is to view each generation as a spatial section of spacetime. A small world then means that a large proportion of space is accessible from $x$ as one moves in the past or future direction, and a small-world phase transition occurs if the accessible proportion increases abruptly to $O(N)$ within a narrow range of generations. To model horizon dynamics as the generation number $m$ varies, we estimate how the size of each horizon predicts those of its neighbors, i.e., how many nodes in generation $m\pm 1$ are connected to at least one of a typical family of $n$ nodes in generation $m$. As explained in Section 2, a reasonable estimate is

[TABLE]

To detect a small world, we study asymptotic horizon size as $m\to\pm\infty$, ignoring slow average decay via fluctuations to zero, as discussed in Section 1.3. This naïve asymptotic size may be estimated by finding a fixed point of (1.4), i.e., a value of $n$ for which $h(n)=n$. A trivial fixed point is $n=0$. Since \frac{d}{dn}\left(h(n)-n\right)\big{|}_{n=0}>0 and $h(N)-N<0$ for $k\geq N$, there is another fixed point strictly between [math] and $N$, i.e., at $n=\gamma N$ for some $\gamma$ between [math] and $1$. Solving for $k$ yields $k=N^{2}\left(1-\left(1-\gamma\right)^{\frac{1}{\gamma N}}\right)$. Qualitatively, $\frac{1}{\gamma}$ estimates how small the small world is in units of the naïve asymptotic horizon size, while $k$ estimates the average connectivity producing this size. The ratio $\frac{k}{N}$ corresponding to a given $\gamma$ tends to $\frac{1}{\gamma}\log\left(\frac{1}{1-\gamma}\right)$ for large $N$. By (1.4), the $m$-fold iterate $H(m)$ of $h(1)$, which estimates $|\sigma_{m}(x)|$, satisfies the Abel functional equation

[TABLE]

where $q:=\left(1-\frac{k}{N^{2}}\right)$, and where $\lim_{m\to\infty}H(m)=\gamma N$ is the naïve asymptotic horizon size. Numerical evidence suggests that $\frac{H}{N}$ approximately maintains its shape and shifts to the right linearly with $\log N$ as $N$ increases. A rough estimate of the $m$-dependent horizon size for all but very large $m$ is

[TABLE]

given by replacing fixed $M$ with variable $m$ in the estimate (1.3) of $P_{N}^{M}(K)$, substituting for $K$ in terms of $k$, and scaling by the naïve asymptotic horizon size $\gamma N$. The left-hand diagram in Figure 5 compares an example of actual relative horizon growth (blue) to the estimate $\frac{H}{N}$ via (1.5) (green) and the rough estimate (1.6) (black, scaled by $\frac{1}{N}$) for $\gamma=\frac{9}{10}$ and $N=1000$. The red line indicates saturation; i.e., inclusion of every node in a given generation. The estimates (1.5) and (1.6) should be viewed with caution since they involve averaging together radically different types of behavior. While some actual horizons grow faster than these estimates, others become trapped at size zero after a few generations. For reasonably large $N$, horizons that survive the first few generations seem to behave similarly to (1.5) and (1.6), except for rare late fluctuations to zero. We may therefore tentatively extrapolate to obtain a rough picture of horizon growth in networks too large to simulate without significant computing resources. The right-hand diagram compares $\frac{H}{N}$ (colors) to (1.6) (black, scaled by $\frac{1}{N}$) for $\gamma=\frac{1}{10},\frac{1}{2}$ and $\frac{9}{10}$, and $N=10^{12}$. Small world phase transitions are early and sharp for large $\gamma$.

More generally, small-world phase transitions may occur due to non-random generation-dependent changes in network structure. Late generations may exhibit different average connectivity and randomness than early generations. For example, adding a dense layer to a convolutional neural network automatically produces a small world. A subtler scenario occurs when network structure changes from random to geometric, or vice versa, over an interval of generations. This induces a change from exponential to power-law scaling of horizons, or vice versa, as mentioned at the end of Section 1.3. In a finite mostly acyclic network, exponential horizon scaling quickly leads to a small world unless the average connectivity is quite low. As explained in Section 3.2, this effect may explain why the early universe appears to be a small world despite inadequate horizon scaling suggested by naïve extrapolation from the present Lorentzian geometric epoch.

2 Mostly Acyclic Directed Networks

We now explain the reasoning behind the estimates in Section 1, and outline methods to extend these estimates to arbitrary mostly acyclic directed networks. This requires generalizing the $M$-generation graphs $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$ to allow generation-skipping edges, generations of different sizes, multiple edges between a given pair of nodes, and small cycles. Some of these generalizations are amenable to tricks such as graph decomposition methods and superposition of edge-addition or removal processes.

2.1 Basic Exponential Estimates

In Section 1.4, we offered large-$N$ estimates (1.3) for the connection probabilities $P_{N}^{M}(K)$ and $\mathscr{P}_{N}^{M}(p)$ between initial and final-generation nodes in $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$. We now sketch an argument supporting these estimates, focusing on $G_{N}^{M}(K)$, the harder of the two cases. We assume that the edge distribution is relatively uniform, meaning that each neighboring pair of generations is connected by roughly $k=\frac{K}{M-1}$ edges. The probability of connection between typical nodes $x_{m}$ and $x_{m+1}$ in generations $m$ and $m+1$ is then roughly $\frac{k}{N^{2}}$, and we assume that $k<<N^{2}$ in the regime of principal interest. Loosely defined, this regime includes the neighborhood of the accessibility phase transition, but does not extend into territory where connection probabilities are indistinguishable from unity and horizons immediately expand to $O(N)$ in either direction from a typical node. Under these assumptions, the probability that $x_{m+1}$ is connected to at least one of a family of $n$ such $x_{m}$ is approximately

[TABLE]

How applicable is this approximation? For exactly $k$ edges between neighboring generations, inclusion-exclusion yields the probability

[TABLE]

The large-$N$ limit sharply favors uniform edge distributions. Approximating each factor $\frac{k-\ell}{N^{2}-\ell}$ in (2.2) by $\frac{k}{N^{2}}$ yields (2.1), which is plausible since $\ell<j\leq n<<N$ in the regime of principal interest. The number of such $x_{m+1}$ connected to at least one of $n$ such $x_{m}$ is then roughly $h(n):=N-N\left(1-\frac{k}{N^{2}}\right)^{n}$, which we used in (1.4) to estimate naïve asymptotic horizon sizes, ignoring slow average decay via fluctuations to zero. We now apply the same idea to recover the estimate (1.3) for the connection probability $P_{N}^{M}(K)$. Setting $n=1$ in (1.4) reproduces the expected degree-$1$ horizon size $|\sigma_{1}(x_{0})|\approx h(1)=\frac{k}{N}$. Feeding this back into (1.4) yields $|\sigma_{2}(x_{0})|\approx N-N\left(1-\frac{k}{N^{2}}\right)^{\frac{k}{N}}$, which has a positive limit when $k=\beta N^{\frac{3}{2}}$ for some $\beta>0$. Dividing by $N$ to obtain a probability estimate reproduces the $3$-generation case of (1.3), to be proven in [10]. Assuming inductively that |\sigma_{M-2}(x)|\approx N\Big{(}1-\exp\big{(}-\frac{k^{M-2}}{N^{M-1}}\big{)}\Big{)} and applying (2.1) again,

[TABLE]

via first-order approximation of the exponent. This expression has a positive limit when $k=\beta N^{\frac{M}{M-1}}$ for some $\beta>0$. Substituting for $k$, (2.3) tends to $1-e^{-\beta^{M-1}}$ for large $N$, reproducing the estimate (1.3), which works reasonably well numerically despite some overestimation. It is worthwhile to pursue finer estimates, however, since even modest overestimation can be significant in applications such as the design of artificial neural networks. Figure 6 illustrates the $5$-generation case, comparing (1.3) (red) to counting functions $F_{N}^{5}(K)$ (blue) for edge-addition processes with $N=50$ and $N=100$.

2.2 Generation-Skipping Edges

We now generalize beyond the case of strictly-graded networks with uniform generation size, first by allowing edges that skip generations. Such edges play an important role in certain artificial neural network architectures such as ResNet [64] and DenseNet [65]. In physics, they represent direct influence from the distant past, which has been proposed as a “moderately nonlocal” phenomenon in causal set theory [66, 67, 68]. Here, we estimate connection probabilities $P(x_{0}\to x_{M-1})$ for initial and final-generation nodes $x_{0}$ and $x_{M-1}$ via superposition of edge-addition processes. Expected horizon sizes may then be inferred in the obvious way. In this context, it is actually easier to work with $G_{N}^{M}(K)$ than $\mathscr{G}_{N}^{M}(p)$. Using the $3$-generation case as an example, random edge addition may be viewed as a superposition of two processes, one adding $2N^{2}$ edges between neighboring generations at relative rate $\frac{2}{3}$, and the other adding $N^{2}$ generation-skipping edges at relative rate $\frac{1}{3}$. To estimate $P(x_{0}\to x_{2})$, we may therefore replace $K$ by $\frac{2K}{3}$ in (1.3) for $M=3$, and by $\frac{K}{3}$ in the exact bipartite connection probability $\frac{K}{N^{2}}$. Since $P(x_{0}\to x_{2})$ is the complement of the probability of connection via neither process, this yields the estimate $P(x_{0}\to x_{2})\approx\left(1-\frac{K}{3N^{2}}\right)\exp\left(-\frac{K^{2}}{9N^{3}}\right)$. Similar reasoning yields estimates for $M$-generation graphs in which edges may skip up to $L$ generations. If $L=M-2$, then edges may connect any pair of generations, and the estimate is

[TABLE]

where the initial factor arises from the possible “direct” edge $x_{0}x_{M-1}$, while the $m$th summand in the exponent arises from edges skipping $m<M-2$ generations. If $L<M-2$, then the estimate is

[TABLE]

In some applications, we may choose to add weights to (2.4) or (2.5) to gradually damp out “longer” edges, for example, to deprecate direct influence from the distant past. Numerical evidence suggests that these formulae inherit modest overestimation from (1.3), but still perform reasonably well for large $N$. Figure 7 compares (2.4) (red) to actual edge-addition processes (blue) for $N=50$ and $M=3$ and $5$.

2.3 Generations of Different Sizes

Next, we generalize to networks with independent generation sizes $N_{0},...,N_{M-1}$. In general, we denote by $k_{mn}$ the number of edges between generations $m$ and $n$ for any $m<n$, but we omit generation-skipping edges at present since they may be easily incorporated via superposition of edge-addition processes. Deep neural networks often exhibit different generation sizes, governed by modeling considerations. In toy examples involving digit recognition using the MNIST dataset [32, 69], a few hundred initial-generation nodes are needed to receive the input data, while just ten final-generation nodes suffice for classification. More serious image-analysis problems require hundreds of thousands of initial-generation nodes and hundreds of final generation nodes, but next-generation applications will increase these numbers significantly. In network-based models of spacetime, increasing generation sizes can represent the familiar notion of spacetime expansion, though other network-theoretic phenomena can also mimic such expansion, as explained in Section 3.2. Returning to the abstract setting, there are $N_{0}N_{1}$ permissible edges in the bipartite case, so the connection probability $P(x_{0}\to x_{1})$ is $\frac{k_{01}}{N_{0}N_{1}}$, and the expected size $|\sigma_{1}(x_{0})|$ of the degree-$1$ horizon of a generation-zero node $x_{0}$ is $\frac{k_{01}}{N_{0}}$. Adding a third generation with $N_{2}$ nodes yields an approximate expected horizon size

[TABLE]

If we assume a roughly uniform edge distribution, meaning that the ratios $\frac{k_{mn}}{N_{m}N_{n}}$ are roughly equal to the fixed constant $p:=\frac{K}{\sum_{m=0}^{M-2}N_{m}N_{m+1}}$, then (2.6) may be rewritten as |\sigma_{2}(x_{0})|\approx N_{2}-N_{2}\big{(}1-p\big{)}^{N_{1}p}. Since $p$ in this expression is the probability that a randomly-chosen permissible edge is included in the graph, we deliberately use the same notation as the inclusion probability $p$ in the model $\mathscr{G}_{N}^{M}(p)$. Dividing by $N_{2}$ and taking the limit as $N_{1}\to\infty$ under the assumption that $p$ is proportional to $N_{1}^{-\frac{1}{2}}$ yields the $3$-generation analogue of (1.3)

[TABLE]

If we assume inductively that $|\sigma_{M-2}(x_{0})|\approx N_{M-2}\left(1-\exp\left(p^{M-2}\prod_{j=1}^{M-3}N_{j}\right)\right)$, then

[TABLE]

Dropping higher-order terms in the exponent, dividing by $N_{M-1}$, and taking the limit as $\prod_{j=1}^{M-2}N_{j}\to\infty$ under the assumption that $p$ is proportional to $\prod_{j=1}^{M-2}N_{j}^{-\frac{1}{M-1}}$ yields the general analogue of (1.3)

[TABLE]

[TABLE]

Evidence is mixed regarding the asymptotic accuracy of (2.9). Figure 8 shows the best and worst estimates (red) for $P(x_{0}\to x_{M-1})$, compared to random edge-addition processes (blue), for ten graphs with $200$ nodes randomly partitioned into five generations. The bad estimate is largely due to the $2$-node bottleneck for $m=1$. Like (1.3), (2.4) and (2.5), (2.9) seems to overestimate the actual probability, but its accuracy may improve as all generations become large. Also, estimates for networks with generation-skipping edges derived from (2.9) via superposition of edge-addition processes may outperform (2.9) itself, since generation-skipping edges can bypass small-generation bottlenecks.

2.4 General Acyclic Directed Networks

Since the node set of any acyclic directed network may be partitioned into generations, our methods are general enough to estimate connection probabilities and horizon sizes for any such network. Such partitions are non-canonical [70], constrained only by the requirement that passage along a directed edge must increase generation number. Physically, partitions are analogous to relativistic frames of reference. Figure 9 illustrates two different partitions of an acyclic directed network. Edges are understood to point up the page. Dashed lines indicate choices of partition.

A node $x$ of a directed graph $G$ is called minimal if no edge begins at $x$, maximal if no edge terminates at $x$, and extremal if it is either minimal or maximal. To generalize our previous results, we focus on connections between minimal and maximal nodes. While $x$ may lose extremal status during an edge-addition process leading to $G$, we are typically interested only in the status of $x$ in $G$ itself, since such a process is usually a contextual tool rather than a realistic description of network genesis. Assuming complete knowledge of the structure of $G$, we describe how to choose generations and permissible edges for estimating connection probabilities and horizon sizes. In applications, we typically lack such complete knowledge, but this is also true for canonically graded networks. The procedure described here merely demonstrates how a general acyclic directed network may be viewed as a graded network. With this in mind, we first use the $K$ edges of $G$ to partition its node set into generations, then forget the edges and perform random edge-addition processes, with permissible edges determined by our choice of generations. Some of these processes produce $G$ when $K$ permissible edges are added, and our methods then yield partition-dependent estimates for connection probabilities and horizon sizes. We typically choose as few generations as possible. Processes to superpose, and weights for these processes, are chosen based on the distribution of generation-skipping edges for each choice of partition. Numerical evidence suggests that these choices often converge to similar results for large networks. However, applications often impose structural constraints that allow for stronger and more specific results. Hence, our goal is not so much to prove theorems about arbitrary networks as to provide tools for analyzing whatever class of networks our applications present us with. Section 3 highlights two such classes.

2.5 Cycles; Acyclic Reduction

Any directed graph $G$ may be reduced to an acyclic directed multigraph $G^{\prime}$, its acyclic reduction, by merging each strongly-connected component into a single node, as illustrated in Figure 10. For discrete spacetime models, this is an example of causal atomic decomposition [53]. More generally, it falls into a broad category of graph decomposition methods called community detection [2]. $G^{\prime}$ may be further reduced to a simple graph $G^{\prime\prime}$, the simple acyclic reduction of $G$, by merging multiple edges. If the strongly-connected components of $G$ are small and few in number, then the connectivity properties of $G^{\prime\prime}$ closely approximate those of $G$. Comparing such properties offers one way to quantify the degree to which $G$ is “mostly acyclic.”

We briefly elaborate on multigraphs such as $G^{\prime}$. In a multigraph, pairs of nodes may be connected by multiple edges in the same direction, as illustrated in Figure 10. Such edges can be important for a variety of reasons. First, they can encode real-world structural nuances in applications; for example, connected pairs of neurons in the mammalian brain often communicate along three to eight different synapses [71]. Second, they are “tolerated” in influential network models such as the configuration model [2], to avoid the bookkeeping nuisance of excluding them artificially. Third, they are equivalent to positive integer edge weights, which may be combined and scaled to approximate real or complex weights in applications such as deep neural networks or discrete quantum systems.

Multiple edges rarely obstruct analysis of network connectivity, since their redundancy can be handled by replacing the number of edges $K$ with the expected number of occupied edge sites $E(K)$. However, passage from $G$ to $G^{\prime\prime}$ can significantly affect connection probabilities and horizon sizes if $G$ has significant strongly-connected components. Merging such components to form $G^{\prime}$ reduces the overall number of nodes, but introduces new nodes of unusually high degree, which may also be extremal. Merging multiple edges to form $G^{\prime\prime}$ modestly reduces the degrees of the new nodes. Overall, $G^{\prime\prime}$ typically has fewer nodes than $G$, more extremal nodes, and higher average node degree, which is no longer random. Extremal nodes of $G$ survive in $G^{\prime\prime}$, so changes to connection probabilities involve only new extremal nodes. These typically have unusually high degree, and are therefore likely to be connected both to each other and to extremal nodes of $G$. Analysis based on $G^{\prime\prime}$ therefore tends to overestimate connection probabilities for $G$. By contrast, horizon sizes may be underestimated because $G^{\prime\prime}$ has fewer nodes. Relative horizon sizes may be either underestimated or overestimated.

Other methods for reducing $G$ to an acyclic directed graph are possible. One may simply delete strongly-connected components, along with all incoming and outgoing edges. This avoids the creation of nodes of unusually high degree, but may drastically reduce connectivity. For example, applying this method to the network from Figure 1 fragments it into a three-component network with much different properties. Alternatively, one may replace each strongly-connected component with a randomly-constructed acyclic “patch” consisting of the same number of nodes and edges. This preserves important parameters of $G$, but reduces connectivity by eliminating “hub-like” effects of strongly-connected components. In particular, it affects connections between extremal nodes of $G$, which are unaffected by passage to $G^{\prime\prime}$. The practical utility of $G^{\prime\prime}$ is determined by how well its connectivity properties approximate those of $G$ in applications. Many interesting networks are very close to being acyclic in this sense. For example, in a citation network, pairs of scientific papers rarely cite each other due to temporal constraints, despite some spectacular opportunities [43, 44], though pending results may be cited in a series of papers [9, 10]. Similar constraints prohibit cycles in most networks of physical events. Spacetime cycles may occur near black holes or at the fundamental scale, but a rigid arrow of time seems to prevail in ordinary regimes. Artificial neural networks can depend crucially on local recurrence [74], but often exhibit a globally feed-forward structure [4], or may be transformed into such a structure via unrolling. Overall, use of $G^{\prime\prime}$ seems useful for a wide variety of applications.

3 Applications

The prominence of network connectivity properties in computer science, biology, social science, and physics leads to many potential applications for the results outlined in this paper. Foremost among these are those involving random network structure, which is non-geometric, non-local, and subject to abrupt changes in connection probabilities and horizon sizes. We focus on two such applications: deep learning in artificial neural networks, and the horizon problem in the cosmology of the early universe. Since machine learning and cosmology appeal to different audiences, we include enough expository material to render the necessary topical facets broadly accessible.

3.1 Deep Learning

In an artificial neural network (ANN) [6, 31, 32], nodes represent artificial neurons, minimal processing units analogous to biological neurons. They convert families of real inputs to real outputs via activation functions. Nonlinear activation functions enable approximation of arbitrary functions of the input variables [72]. Directed edges represent artificial synapses, connections between artificial neurons. We focus on acyclic networks, called feedforward networks, though recurrent neural networks (RNNs) offer advantages such as Turing completeness [73] and efficient sequence modeling [74]. Generations in an ANN are called layers. Non-input layers are often specified along with their incoming edges, allowing modular description of network architectures in terms of layer types such as dense, convolutional, and pooling layers. While biological brains remain far more complex [75], ANNs mimic enough of their properties to play a central role in modern artificial intelligence.

Deep neural networks. Figure 11 illustrates a deep neural network (DNN), characterized by the presence of deep or hidden layers between its inputs and outputs. Deep layers reduce the number of parameters necessary for accurate function approximation [76]. They also enable accessibility phase transitions and small-world phase transitions, with consequences described below. The illustrated DNN has $M=5$ layers, labeled $m=0$ to $4$, though typically only deep layers are counted. A left-to-right convention for information flow is common, but we maintain our spacetime-inspired “up the page” convention. Each edge in the network is assigned a real weight, which determines strength of influence between its initial and terminal neurons.

Network architectures: dense, convolutional. The network in Figure 11 is dense, meaning that each non-input neuron is connected to every neuron in the previous layer. The network is therefore non-geometric and non-local. It has the structure of a strictly graded graph of the type studied in Section 2.3, with independent generation sizes $N_{0},...,N_{M-1}$, and with every permissible edge included. These edges are shown in gray; colors are for future reference. Contrasting with this dense architecture are convolutional neural networks (CNNs) [36, 37, 38], inspired by biological vision systems [77], which are highly local and sparse, with each neuron connected to only a small kernel of neurons in the previous layer. More generally, the term “CNN” is used to describe any DNN that makes significant use of convolutional layers. CNNs are the most successful DNNs in applications, and are central to our discussion here. However, many other architectures are possible, including RNNs and even networks of networks [78]. Absolute connectivity properties of DNNs have traditionally remained fixed following design, though subsequent alterations such as pruning have recently become common [79, 33, 34, 35, 39]. However, functional connectivity is determined not only by network topology, but also by edge weights, which change as the network learns. This leads to interesting horizon dynamics.

Applications; feature extraction. DNNs excel at qualitative pattern recognition involving complicated data sets. Famous examples include image analysis [38], speech recognition [80], vehicle navigation [81], and game strategy [82, 83]. Such problems require the network to extract, synthesize, and interpret definitive features of its input data. Different network layers often perform different tasks in this context. For example, early convolutional layers in a large CNN might convert information to geometry by extracting simple local features from neighboring pixel values in an image, while late dense layers might convert geometry to information by synthesizing complex features into overall shapes and classifying them. A useful toy image analysis problem, already mentioned briefly in Section 2.3, is to classify handwritten digits from the MNIST dataset, encoded as grayscale images [32, 69]. Features in this context consist of lines, curves, and loops.

Network training: parameter space, gradient descent. Input data for a DNN is converted into numerical tensors for processing. Most such data is irrelevant for tasks such as image classification, but discriminating between relevant and irrelevant data has been one of the principal long-term challenges of artificial intelligence. DNNs address this challenge via a trial-and-error training process to minimize a loss function $L$, which measures the error of actual outputs for training samples with known “correct” outputs. The arguments of $L$ are the parameters characterizing the network, which consist of the edge weights and possibly a few other quantities. After each training batch, the parameters are updated and the process is repeated. Analytical minimization of $L$ is typically infeasible due to the large number of variables involved, which is why numerical trial and error is used. Gradient descent and its variations are the principal methods. In gradient descent, the graph of $L$ is regarded as a hypersurface in parameter space, whose downward slope $-\nabla L$ is followed step-by-step to find a local minimum. The step size used for each parameter update is called the learning rate. For simplicity, we call the family of local minima the solution space, though solutions are only approximate. However, local minima can yield surprisingly accurate results for a well-designed network. For example, a modest digit-recognition DNN can achieve $99\%$ classification accuracy. Training is successful if the DNN can generalize, i.e., accurately process previously-unseen inputs.

Dimension reduction; pruning; phase transitions. Weights in a trained DNN can be highly correlated, which implies that the solution space is concentrated near a relatively low-dimensional submanifold of parameter space. This dimension reduction phenomenon is common in applications. The simplest type of correlation occurs when many weights approach zero. Training methods explicitly favoring small weights, such as Tikhonov regularization [31], have produced favorable results. In principle, arbitrary correlations may be re-expressed in this manner via coordinate transformations. Remaining edges with significant weights then constitute a sparse residual network, like the one illustrated in black and red in Figure 11. From this viewpoint, training induces a functional edge-removal process. Pruning, inspired by biological synaptic pruning, may also be applied to eliminate superfluous edges entirely [33, 34, 35, 39]. Both processes can induce accessibility phase transitions similar to those induced by random processes, with potentially important consequences. Since they involve edge removal, such transitions are time-reversed, manifesting as abrupt decreases in functional connectivity. Topological phase transitions are less relevant in this context, since functional fragmentation of a DNN is possible only for specialized datasets that can often be separated a priori. By contrast, a broad variety of training scenarios can induce accessibility phase transitions.

Locality; sparseness; physics connections. While empirical dimension reduction offers ample justification for studying such transitions, low-dimensional solution spaces are also favored on theoretical grounds. Recent work linking deep learning to Hamiltonian mechanics [85] systematically demonstrates how basic structural properties such as symmetries and locality lead to functionally sparse networks. Applications often incorporate such properties for physical reasons. For example, photographic images inherit locality from the physical space of their subjects. Training a network for image classification therefore encourages sparseness, which can induce an accessibility phase transition if sufficiently pronounced. Training processes for which such a transition predicts optimization of the loss function $L$ may be loosely characterized as satisfying a descent-transition criterion. Training-induced transitions may manifest differently than those induced by random processes. Functional sparseness may appear in some layers while others remain relatively dense, and transitions may occur in subnetworks rather than globally. Also, as discussed below, networks may be designed to anticipate structural properties that might otherwise have arisen naturally via a phase transition. For example, CNNs leverage built-in locality, rather than “rediscovering” it during training. Layers that might otherwise have become sparse are sometimes chosen small enough to remain relatively dense, or are pruned down to a small size during training. A compelling reason to limit layer size is to avoid overfitting, in which a network ascribes undue significance to artifacts of its training data.

Applied horizon dynamics: exceeding biological limitations. Training is often the “bottleneck” for deep learning applications, since it is time-consuming and computationally expensive. For example, Google DeepMind’s overnight training of its multipurpose DNN program AlphaZero to play superhuman chess required an immense hardware array involving 5000 tensor processing units [82]. An open-source analogue called Leela Chess Zero has required more than a year to achieve comparable results with more modest resources [86]. Our principal motivation for applying phase transition theory to deep learning is to reduce training cost by improving network design and training diagnostics. We focus first on design. A key objective is to reduce the initial parameter space dimension without sacrificing performance, but this can be difficult, tantamount to partially solving the deep learning problem itself. Established approaches include feature engineering to jumpstart the sorting process prior to input, and specific sparse designs such as CNNs to pare down a larger naïve parameter space. A limitation of both approaches is reduced generality and expressive power. Recently, significant progress has been made via connectivity-enhancing notions such as generation-skipping edges [64, 65] and expander graphs [21, 40, 41]. We propose a general approach that leverages the horizon dynamics of hybrid local/random networks. This approach complements the CNN-exemplified method of “mining” biology for design clues by deliberately exploiting structural features that biological systems cannot employ for physical reasons. For example, the human brain cannot utilize the nonlocal connectivity properties and rapid horizon growth of random networks, due to its embedding in three-dimensional space.

Optimizing sparseness. We begin our discussion of DNN design by posing a general question: how sparse can a DNN with specified layer sizes be without significantly reducing its training potential for interesting problems? A biological motivation for sparseness is the recent observation that only a small proportion of neurons in the auditory cortex fire in response to any given stimulus [87, 88]. Perhaps surprisingly, the idea of constructing general sparse network topologies a priori, rather than achieving sparseness by pruning denser networks, seems to have been implemented only very recently, with such architectures as X-Nets [40] and RadiX-Nets [41]. An obvious condition limiting sparseness is that each output neuron must be sensitive to all, or at least most, of the input data, and it is desirable in practice to have relatively high connectivity even at shorter ranges. Connection probabilities and horizon sizes are therefore of central interest, especially since highly-connected sparse networks like those mentioned above have already produced accurate results with good efficiency [39, 40, 41]. A lower bound for acceptable sparseness in random networks is given by the accessibility phase transition, since sparser random networks exhibit low connectivity. However, this leaves room for networks significantly sparser than biological neural networks or typical non-random ANN architectures. In particular, using randomness as a deliberate design feature, in the same spirit as the pseudorandom X-Net designs, offers potentially decisive advantages. As we demonstrate below, hybrid local/random networks can exceed even random networks in connectivity, while preserving the local feature-extraction potential necessary for applications such as image classification.

Background: dense layers in a CNN. We next highlight the cost of using dense layers near the output in a large CNN, a common design method. Figure 12 illustrates past and future horizon growth in such a network, loosely based on the influential VGG network [89]. Progress in DNN construction is so rapid that the original VGG designs have been surpassed, but these designs still provide useful comparisons since they have been so deeply analyzed. Each convolutional layer in our network is arranged to form a three-dimensional “spatial section” of the network, where two dimensions involve actual spatial geometry of input images, and the third represents different filters used for feature extraction. The entire network is therefore four-dimensional, with a single dimension of process time pointing up the page. Like Minkowski spacetime diagrams, these diagrams suppress two “spatial” dimensions. Each cell in the lower (red) regions represents a neuron in a $1$-dimensional cross-section of a convolutional layer. Cells grow larger near the top of these regions to represent pooling, whereby activation functions are combined over small patches of neurons. Red lines indicate pooling layers. Blue regions indicate stacks of dense layers, used to synthesize, interpret, and classify combinations of features extracted by the convolutional layers. We focus for the moment on horizon growth in the convolutional layers, looking forward in process time from the input layer and backward from the final pooling layer. The top cross section in the right-hand diagram consists of two cells rather than one, since the final pooling layer combines these cells prior to the first dense layer. While there is reasonably high connectivity between the input layer and the final pooling layer, shorter-range connectivity between pairs of convolutional layers is rather poor. This limits early exchange of information and leaves the entire synthesis process to the dense layers.

We now pose a second question: can the synthesis process in a large CNN be partially superposed with the extraction process, allowing reduction or elimination of one or more dense layers? Recent results suggest that the answer is affirmative [39, 40, 41]. The reason for this question is that dense layers are costly in terms of parameter space dimensions. To be specific, the final pooling layer in configuration D of the VGG network has $7\times 7\times 2^{9}\approx 2.5\times 10^{4}$ neurons, due to the use of $2^{9}$ filters in the final convolutional layer [31, 90]. The first dense layer has $2^{12}$ neurons, so the number of connections between the two is roughly $10^{8}$, contributing more than $74\%$ of all network parameters. The remaining dense layers contribute a further $15\%$. This strongly incentivizes replacing dense layers with sparser structure. We mention four out of many possible methods to accomplish this. First, one may simply remove a large proportion of edges in the dense layers, leaving behind sparse random layers, similar to pseudorandom X-Linear layers [40]. The success of various pruning and dropout methods hints at this approach. Second, one may employ non-random sparse architectures, as in RadiX-Nets [41]. Third, one may employ fuzzy or smeared kernels to obtain a hybrid local/random structure, in which connections to a given neuron are locally concentrated, but not sharply cut off outside a hyperrectangular window. Fourth, as a special case of the third method, one may augment a pure CNN by superposing sparse random structure. We illustrate a toy example of this method below. In general, such methods may also incorporate generation-skipping edges, as in ResNet [64] and DenseNet [65].

Example: augmented CNNs. Figure 13 illustrates the effect of augmenting toy CNNs by superposing sparse random structure. In the left-hand diagram, the CNN has three layers (four generations, including input), while in the right-hand diagram it has five layers (six generations). Each generation has $28\times 28=784$ neurons, the input size for MNIST digit recognition. The kernel size of each CNN is $3\times 3$, and the stride length is $1$. The total numbers of permissible edges are about $1.8\times 10^{6}$ and $3.1\times 10^{6}$, but the CNNs have only about $2\times 10^{4}$ and $3.4\times 10^{4}$ edges, with $6\%$ and $15\%$ connectivity between initial and final generations. Green curves show the connection probability estimates (1.3) for random edge-addition processes with the same generation sizes, beginning with zero edges. Blue curves snapshot actual connectivity for two such processes, measuring the proportion of connections between a central initial-generation neuron and the final layer. Edge and corner neurons behave similarly. Horizontal lines indicate $10\%$, $50\%$, and $90\%$ connectivity. For example, in the left-hand diagram, $K\approx 2\times 10^{4}$ random edges produce about $50\%$ connectivity. Accessibility phase transitions occur near $K\approx 2\times 10^{4}$ and $K\approx 1.5\times 10^{4}$. Red curves show the effect of superposing the same edges on the CNNs, forming hybrid local/random structures.

Figure 14 shows a $2$-dimensional cross section of the right-hand hybrid network from Figure 13 after a few hundred random edges are added. The shaded triangle illustrates horizon growth in the original CNN, while the black nodes illustrate the faster horizon growth in the hybrid network. Adding about $2500$ random edges increases the connectivity from $15\%$ to $95\%$, even though the independent connectivity of these random edges is negligible. A single dense layer, meanwhile, costs roughly $6\times 10^{5}$ edges.

Hybrid synergy: more connected and more local. Simulations reveal a fortuitous synergy for large networks: $K$ local/random edges can produce better connectivity than $K$ random edges, even though purely random structure offers much better connectivity than purely local structure. Hybrid design therefore offers a “cake and eat it too” possibility for combining the structural advantages of convolutional and dense layers. The left-hand diagram in Figure 15 compares a random network (blue) to a hybrid local/random network (red) with the same number of edges. Each curve shows the proportion of connections between a central first-generation neuron and the final generation. The random network is constructed by adding $40000$ random edges to $10$ layers ($11$ generations, including input) of $784$ nodes each. The hybrid network is constructed by first adding $7840$ edges in a minimal convolutional structure with kernel size $1$, then adding the final $31160$ edges from the random network. The hybrid network requires only half as many edges to reach $100\%$ connectivity, while retaining local properties absent in the random network.

The left-hand diagram in Figure 16 illustrates the type of structure driving the rapid hybrid growth of connectivity in Figure 15. Kernel size $1$ means that each neuron is connected to the corresponding neurons in neighboring generations. Vertical flow of information is therefore assured, while superposed random edges open new conduits for this flow. The comprehensive superiority of the hybrid edge-addition process prompts an interesting open optimization question: what edge-addition process optimizes connectivity growth for given layer sizes? The right-hand diagram in Figure 16 shows an extreme case in which $O(N+M)$ edges produce $100\%$ connectivity for $M$ generations of $N$ neurons each, demonstrating that the $O(N^{\frac{M}{M-1}})$ edges required for similar connectivity in (1.3) can be significantly improved. However, this is an impractical “short circuit” design. We therefore restrict attention to homogeneous processes like those in Figure 15, which treat all parts of the network equally, excepting possible edge effects. Such processes allow for networks with broad general training potential. A similar emphasis prevails in the fundamental physics context discussed in Section 3.2, due to the empirical uniformities of spacetime at measurable scales. We remark that maximizing input-output connectivity need not maximize typical horizon growth, as illustrated by the “short-circuit” network in Figure 16. This network achieves high connectivity at low cost by spending only a few edges on intermediate layers, while typical nodes have empty horizons. Such pathologies are less of an issue for homogeneous edge-addition processes.

The right-hand diagram in Figure 15 compares the results of $20$ edge-addition processes for random (blue) and hybrid (red) networks. Random networks are constructed by adding $5000$ random edges to $10$ layers ($11$ generations) of $100$ nodes each, while hybrid networks combine a minimal convolutional structure of $1000$ edges with the final $4000$ edges from the corresponding random networks. Two features stand out in this diagram. First, the hybrid networks typically require only about half as many edges to reach $100\%$ connectivity as their random counterparts. Second, connectivity for individual neurons increases in a smoother and stabler manner in the hybrid networks, since the vertical information flow ensured by the minimal convolutional structure “preserves the gains” in horizon size due to nonlocal connections in the random structure. As a result, while $4$ out of $20$ random networks have zero connectivity between the chosen neuron and the final generation after $2000$ edges are added, all of the hybrid networks with at least $2000$ edges have at least $70\%$ connectivity.

Future applications: large populations. The toy CNNs in Figures 13 through 16 have uniform layer size, unlike typical large CNNs such as VGG, whose layer sizes shrink by hundreds of times between input and output layers. For such networks, reducing late dense layers can decrease the number of edges by perhaps one order of magnitude. However, future applications such as recognition of individuals among a large population may require at least as many outputs as inputs. In this case, reducing late dense layers might decrease the number of edges by thousands of times or more. While ethical use of such technology cannot be overemphasized, improved efficiency can at least democratize its use. Due to weight sharing in CNNs, superposed random edges may contribute relatively heavily to parameter space dimensions, but this effect is unlikely to offset huge overall decreases in edge counts. Further data is needed to determine how such sparse hybrid networks will perform compared to their denser conventional counterparts, but results involving similar architectures are encouraging [39, 40, 41].

Phase transitions and training diagnostics. We now sketch how phase transition theory might contribute to training diagnostics, addressing issues such as management of learning rates, detection of peak performance, and avoidance of “bad” minima of $L$. Design improvements might reduce the scope of such analysis by anticipating transition-induced network properties, but the resulting methods might still apply to problems for which optimal design is unknown. For brevity, we focus on training processes that satisfy a descent-transition criterion, already mentioned above, in which an accessibility phase transition heralds the approach of a local minimum of $L$. The benefit of this scenario is that it allows direct application of knowledge about network structure, unmediated by $L$. For example, candidate stopping points for training might be detectable in terms of average connectivity properties, reducing the role of costly performance-based testing procedures. Learning rates might be adjusted similarly. Methods such as batch normalization [91] and layer normalization [92] offer precedent for such direct analysis and manipulation of network properties.

Small-world behavior; spacetime analogy. We conclude this section by discussing small-world behavior in DNNs. This leads to a profound physical analogy between DNNs and spacetime, foreshadowed in the context of general networks in Section 1.4. Here, “time” is the process time pointing from inputs to outputs, while each layer of the DNN represents a spatial section of spacetime. For some networks, such as the convolutional layers in Figure 12, this analogy is obvious, since such layers encode actual spatial information. Other networks, such as the dense network in Figure 11, are dissimilar to familiar spacetime, since they are non-geometric and non-local. Small-world behavior depends on the interplay between local and nonlocal structure, with local structure favoring largeness and nonlocal structure favoring smallness. Transition to a small world may occur in either direction of process time: in the future direction in Figure 12, and in the past direction in Figure 11. As mentioned above, the distinction between these opposite structural trends may be conceptualized as a contrast between converting geometry to information and converting information to geometry.

We spotlight small-world behavior in a dense network like the one in Figure 11, both before and after training. This behavior is not universal, but foreshadows the results of the next section in a useful way. Assuming random initialization of weights, small-world phase transitions in both directions of process time are immediate prior to training, since neighboring layers exhibit many random functional connections. This changes as the network is trained and weights become correlated, leading to functionally sparser structure. Red nodes and edges in Figure 11 illustrate horizon growth looking backward in process time from an output neuron $x$ in the trained network. Concretely, this means tracing data arriving at $x$ back to its origins in the input layer. Near the output layer, data is sufficiently sorted that $x$ receives significant input from only a few of its immediate predecessors. If the network is trained for classification, this means that identification of a given cross section of input data as an instance of “$x$” depends on a limited number of high-level features. The network therefore exhibits locality in its final layers, and the world remains large. Past horizons of $x$ occupy only a small proportion of these late layers, exhibiting relatively slow power-law growth. Near the beginning of process time, however, this behavior breaks down. Data on its way to $x$ is not yet significantly sorted, instead remaining distributed throughout the early layers. In structural terms, many different low-level features contribute to a typical high-level feature. Naïve extrapolation of horizon growth near the output layer then leads to the erroneous conclusion that only a small proportion of input data influences $x$. This deceptive appearance based on local horizon growth bears an unmistakable similarity to a well-known physical problem examined in the next section.

3.2 Cosmological Horizon Problem

We now reexamine a famous cosmological problem, whose implications may support the novel hypothesis that spacetime itself emerges from network structure at the fundamental scale. Emergence means that the familiar smooth four-dimensional Lorentzian geometry of general relativity is really just a low-resolution approximation, analogous to the illusion of smooth imagery on a pixelated display. Though we do not yet know exactly how this particular geometry came to dominate at large scales and late times in our universe, such emergence naturally favors a change from exponential to power-law scaling of horizons, since random structure a priori dominates the primordial sum over evolutionary pathways that generalizes Feynman’s path integral in quantum gravity. Standard astronomical data suggests that such a scaling change did in fact occur in the early universe, a conclusion broadly acknowledged on empirical grounds despite absence of a compelling mechanism to account for it in conventional relativistic cosmology and quantum field theory [46, 47, 57, 93]. This conclusion follows from evidence of interaction between regions of spacetime that appear causally disconnected on the basis of naïve backward extrapolation of local Lorentzian horizon growth, first pointed out by Rindler [42] in the middle 1950s. Foremost among this evidence is the large-scale homogeneity of the observable universe, which may be inferred, under modest assumptions, from measurements of the cosmic microwave background (CMB) [43, 44, 93]. The consensus explanation for this homogeneity is equilibration via a mixing process, since otherwise the properties of the early universe would appear to be coincidentally fine-tuned. However, mixing requires causal contact, and therefore much faster horizon growth than what is observed today. This clash between extrapolated and empirically-inferred degrees of interaction is called the horizon problem.

General horizon problem. Figure 17 shows a schematic description of the horizon problem, independent of specific structural assumptions except for a beginning of time and an absence of large-scale cycles. Possible “cosmological prehistory” such as eternal inflation is not included in this picture. Events $y$ and $z$ are ancestors of event $x$, lying on its past horizon. In relativity, the past horizon of $x$ means the boundary of the set of events than can influence $x$, called its causal past. By contrast, our horizons $\sigma_{m}(x)$ from Section 1.3 are defined to be cross sections of the past or future of $x$ rather than boundaries, because the latter are harder to generalize to non-geometric settings. Here, either notion suffices, due to a judicious choice of $y$ and $z$. The apparent locations of $y$ and $z$ (white nodes) from the perspective of an observer at $x$ may differ from their actual locations due to factors such as expansion or contraction of spacetime, gravitational lensing, variation of the speed of light, and spatial anisotropy. Since $y$ and $z$ share common ancestors such as $w$, it is not surprising if they exhibit common environmental features indicative of equilibration. However, such features would surprise a naïve observer at $x$, unaware of the drastic stretching of horizons near the beginning of time. Such an observer would assume power-law horizon scaling derived from the local spacetime dimension and speed of light, yielding disjoint apparent horizons (dashed lines), and would therefore recognize a horizon problem.

Inflationary hypothesis. Horizon stretching in the schematic diagram of Figure 17 suggests a profound early change in the speed of light, a dark-horse mechanism for resolving the horizon problem [94, 95]. However, the dynamical Big-Bang model of spacetime genesis admits a different mechanism involving a brief, drastic period of spacetime expansion. This is the inflationary hypothesis of Guth [46] and Linde [47], illustrated in Figure 18, which has dominated mainstream cosmology since 1980. It posits that expansion spiked in an explosive inflationary epoch lasting roughly $10^{-32}$ seconds, caused by a mysterious inflaton field, which increased local spatial volumes by a factor of $10^{25}$ or more. Afterward, inflating spacetime decayed into ordinary spacetime, with a much slower expansion rate. Looking back in time from $x$ under the inflationary paradigm, the actual horizons of $y$ and $z$ are much smaller than their apparent horizons, yet overlap in the crowded environment near the beginning of time. Looking in the opposite direction, adjacent regions inside this overlap are blasted far apart during the inflationary epoch, producing large but widely-separated futures. Adjusted apparent pasts of $y$ and $z$ (dotted curves), based on ordinary spacetime expansion inferred from red-shift data, are also disjoint, illustrating that the horizon problem is not merely a failure to account for obvious processes. Horizon stretching near the top of the diagram represents the standard view that spacetime expansion resumed gradual acceleration around 5 billion years ago.

Among other successes, inflation also addresses the flatness problem, which is the question of why spacetime seems fine-tuned to exhibit so little large-scale curvature, and offers a possible mechanism for structure formation in the early universe via quantum fluctuations. However, it also faces serious objections. The hypothetical inflaton field requires such fine-tuning that reasonable inflationary scenarios may be less likely than coincidental homogeneity and flatness. Recent astronomical data disfavors simple inflationary models [60, 61, 62, 63]. Perhaps most troubling is the lack of motivation and context for the hypothesis. It serves as an ex post facto patch for relativistic cosmology, extrapolating familiar spacetime geometry into a regime involving energies and interactions beyond present knowledge. The identity, origin, and overall behavior of the inflaton field remain mysterious. As described by Ellis, Maartens, and MacCallum [57], inflation “remains a phenomenological scenario that is yet to be rooted in a fundamental theory.”

Network-theoretic small-world alternative: By contrast, spacetime emergence from network structure naturally favors horizon scaling of the type needed to resolve the horizon problem. Assuming a pre-geometric epoch dominated by random network structure, exponential horizon scaling follows automatically by the reasoning described in Section 1.3. Feynman’s path summation approach to quantum theory [59] motivates such a pre-geometric epoch, as we now describe. In ordinary quantum mechanics, Feynman’s approach predicts the probabilistic behavior of a particle by integrating the complex exponential $\exp\left(\frac{i}{\hbar}\mathscr{S}(\gamma)\right)$ over all permissible particle trajectories $\gamma$, where $\mathscr{S}$ is the classical action, $\hbar$ is Planck’s reduced constant, and $i$ is the imaginary unit $\sqrt{-1}$. Most trajectories deviate wildly from the classical trajectory $\gamma_{\textnormal{CL}}$, but these destructively interfere, effectively reproducing $\gamma_{\textnormal{CL}}$ for macroscopic particles. However, random effects are significant for small particles. In quantum gravity, the path integral is replaced by a sum over evolutionary pathways for spacetimes, which may be modeled via geometries, spin foams, causal dynamical triangulations, or in the present case, networks. Every permissible process of network evolution contributes to this sum. Networks closely approximating classical relativistic geometry play the role of $\gamma_{\textnormal{CL}}$, but these are vastly outnumbered by random networks. Hence, random effects such as exponential horizon scaling tend to dominate a small network-theoretic universe near the beginning of time.

Figure 19 illustrates the effect of a pre-geometric epoch on past horizon growth in a network toy model with $20$ generations of $60$ nodes each and an average of $120$ edges between neighboring pairs of generations. Edges are suppressed for clarity. The shaded horizontal band indicates a brief semi-geometric epoch during which network structure changes from random to geometric. The discussion of hybrid local/random structure in Section 3.1 suggests that semi-geometric horizon growth might exceed even random horizon growth, but the figure does not show this possibility. For illustrative purposes, only two dimensions are shown in the geometric epoch. Each node in this epoch is connected to exactly two previous-generation nodes. This represents the chosen rate of information exchange, analogous to the speed of light, and independent of the dimension. In the pre-geometric epoch, a total of $8\times 120=960$ non-generation-skipping edges are distributed randomly. Black nodes indicate past horizon growth for a final-generation node $x$. Growth is linear with temporal depth in the geometric epoch because the spatial dimension is $1$, but increases drastically to exponential growth as one moves back in time through the semi-geometric epoch to the pre-geometric epoch. Here, the network becomes a small world, and further horizon growth is arrested by its overall size. In the pre-geometric epoch, thorough mixing occurs within a few generations, resolving the horizon problem.

Lorentzian networks. The toy model in Figure 19 differs importantly from actual spacetime due to the preferred frame of reference defined by its generational structure. Its geometric epoch is therefore approximately Euclidean rather than Lorentzian. Even the network in Figure 9, which has no preferred frame, is still nearly Euclidean, with some frames “more equal than others.” Lorentzian networks are not used in these figures for the prosaic reason that they make for awkward illustrations, but their qualitative horizon growth properties are the same as for other geometric networks. The left-hand diagram in Figure 20 shows the shape of an approximately Lorentzian network, with edges suppressed for clarity. Edges attach each node $x$ to a family of nodes located just inside an imaginary “light cone” with $x$ at its apex, and therefore appear to nearly coincide in drawings, as shown in the right-hand diagram. As explained in Section 2.4, such a network may be partitioned into generations noncanonically, which amounts to choosing a frame of reference. For any such choice, a large proportion of edges skip generations. Since nature favors this type of network architecture, it is interesting to transport it into other settings, for example, artificial neural networks. Unfortunately, the term Lorentzian neural network has already been coined to mean something entirely different [96].

Emergence: causal metric hypothesis. Small-world resolution of the horizon problem, as we have proposed it, depends on the hypothesis that spacetime can emerge from network microstructure. This is far from obvious; for example, strictly graded graphs such as $G_{N}^{M}(K)$ and $\mathscr{G}_{N}^{M}(p)$ are almost certainly ruled out due to experimental constraints on Lorentz invariance violation [97]. More promising are causal network models, in which nodes represent events, and directed edges represent causal relationships between pairs of events. After foreshadowing by Finkelstein [98, 99], Hawking [100] and Malament [101] proved in the late 1970s that many Lorentzian geometries may be described simply in terms of causal structure and scale data. It was soon understood that such scale data, represented by the conformal factor in the Lorentzian metric, comes for free in the discrete context, via enumeration of fundamental spacetime elements [50, 102, 103]. This suggests that spacetime geometry can emerge from causal network structure. We refer to this idea as the causal metric hypothesis [53, 54, 55]. Popular among causal network models are causal sets, developed by Sorkin et al. [50], defined by transferring relativistic notions to the discrete context with as few changes as possible. Physically-relevant causal sets, and other types of causal networks, may be constructed by randomly sprinkling elements into a pre-existing Lorentzian manifold $X$, defining edges via its causal structure, then erasing $X$. This method is used in Figure 20, where $X$ is flat Minkowski spacetime. A technical distinction is that causal sets are defined to be transitive, which inconveniently obscures the distinction between direct and indirect causation. Our causal networks are not assumed to be transitive.

Example: trillion-node generations. The left-hand diagram in Figure 21 illustrates the difference in horizon growth between large four-dimensional sprinkled causal networks and pre-geometric networks, using (1.4) as an approximation. Curves correspond to networks with $10^{2}$ generations of $10^{12}$ nodes each, and average node degrees $1.5$ (blue), $3$ (yellow), and $1000$ (green). The abscissa is the generation number, and the ordinate is the base-$10$ logarithm of the ratio of horizon sizes for the pre-geometric and geometric cases. Even for average degree $1.5$, the horizon size predicted by (1.4) is nearly $10^{6}$ times larger than the geometric horizon size after about $75$ generations, while average degree $1000$ yields a horizon size nearly $10^{10}$ times larger after just a few generations. The eventual decay of each curve is due to small-world saturation, i.e., halting of exponential horizon growth due to limited network size. This corresponds to thorough mixing in the context of the horizon problem.

Preferential attachment. Rapid horizon growth in the early universe may be further enhanced by a “rich-get-richer” mechanism called preferential attachment or accumulated advantage [2], whereby nodes that appear early in a network growth process, or with many incident edges, are likely to possess the most incident edges late in the process. For example, early-written papers are more likely to accumulate many citations than late-written papers, because there are more opportunities for them to be cited, and more existing citations to advertise them. The left-hand diagram in Figure 22 illustrates a simple process leading to a weak form of preferential attachment. Nodes are added sequentially, each attached by a single edge. The resulting network is a random tree. Early nodes such as $1$ and $3$ tend to acquire more edges than later nodes. Many other network growth processes exhibit similar behavior, often more marked; for example, individuals with many social connections are likely to acquire more connections rapidly, regardless of chronology. Preferential attachment is possible but not automatic in a causal network. If the random tree in Figure 22 is interpreted causally, then each event has a unique cause but may produce multiple effects, which seems unrealistic. Allowing multiple causes introduces a balancing effect, since later nodes have more possible attachment sites. However, there may not be an absolute balance, on average, between the number of causes and effects of events in the real world. Idealized Lorentzian geometry is locally time-symmetric, but we know that real physics discriminates between cause and effect due to the second law of thermodynamics. Even a small asymmetry of this type leads to significantly different early horizon growth. The right-hand diagram in Figure 21 compares horizon growth for different average node degrees, using the same random networks involved in the left-hand diagram. For example, the blue curve shows that doubling the average degree from $1.5$ to $3$ produces a horizon more than $10^{7}$ times larger after about $25$ generations. An analogous change in a geometric network merely doubles horizon sizes, which demonstrates that the potential effect of preferential attachment is vastly amplified in a pre-geometric epoch.

Problem of time; sequential growth. The significance of preferential attachment in resolving the horizon problem depends on the role of network growth processes in spacetime dynamics. This is distinct from the more basic question of emergence, and involves the so-called problem of time. Though relativity has standardized geometric unification of space and time, there remains deep confusion about how the two differ, and whether or not past, present, and future should really be accorded equal ontological status. Reducing the distinction to a sign difference in the metric signature trivializes the issue by simply canonizing the time-symmetric relativistic viewpoint, leaving no obvious role for time-asymmetric growth-related mechanisms such as preferential attachment. Quantum gravity then involves summing over a configuration space of fixed networks. The right-hand diagram in Figure 22 shows a small part of such a space, with member networks appearing inside large circles. Connections among these circles are for future reference. This summation procedure admits no intrinsic distinction between past and future directions, since each network in the configuration space has a dual network given by reversing its edge directions.

By contrast, one may choose to regard network models of spacetime as evolving, rather than fixed, objects. This is how Sorkin and Rideout approach the problem of time in their sequential growth dynamics for causal sets [104]. In sequential growth dynamics, a causal set develops one node at a time, much like the random tree in the left-hand diagram in Figure 22. Similar processes apply to causal network models. A higher-level edge may be defined between a pair of networks $A$ and $B$ whenever $A$ can evolve into $B$ by acquiring a single additional node and its accompanying edges. The right-hand diagram in Figure 22 illustrates such higher-level edges as connections between large circles. Recognition of such relationships transforms the configuration space into a network of networks, in which past and future directions are intrinsically distinct [53]. Quantum gravity then involves summing over network growth processes instead of individual networks. Bold circles and edges illustrate such a process. One may then attempt to specify how networks are likely to develop, rather than interpreting each network as a fixed unit. The resulting evolutionary rules may or may not favor preferential attachment, depending on the details of the model.

Spacetime $\leftrightarrow$ DNN analogies. We close with a brief discussion of analogies between our two application topics: deep learning, and the cosmological horizon problem. Throughout the paper, we have used spacetime-inspired conventions and heuristics to describe general network structure, and the fact that popular DNN architectures such as CNNs incorporate geometric properties inherited from applications strengthens the conceptual connections between the two. At the end of Section 3.1, we explicitly compared a trained dense DNN to spacetime, anticipating our analysis of the horizon problem. Conversely, one may compare spacetime to a DNN, and this leads to a variety of interesting, though speculative, ideas. Unsurprisingly, there has been a recent explosion of interest in connections between fundamental physics and machine learning, involving such topics as the renormalization group [105], quantum field theory [106], and the intersection of machine learning and quantum computing [107]. The more radical idea of comparing spacetime itself to a DNN is not without historical precedent in the “physics as computation” tradition [98, 108, 109].

Beginning from a network-theoretic spacetime model such as causal set theory, it is not difficult to incorporate the remaining characteristics of an individual DNN: processing is similar to path summation, edge weights to a Lagrangian, classical action, or phase map, and activation functions to an operator field. More tenuous, but still interesting, are comprehensive cosmological analogies attempting to identify universal counterparts to other aspects of machine learning. For example, the discussion at the end of Section 3.1 suggests comparing spacetime to a trained DNN, since early small-world behavior reflects broad distribution of information near the input layer before localization due to sorting. An obvious question is how to reconcile this sorting viewpoint with the second law of thermodynamics in a network-theoretic setting [110], but we cannot analyze this topic here. Biological synaptic pruning offers an interesting analogy related to the horizon problem: high random connectivity in the early universe may be compared to excess connectivity in an immature brain, and structural changes through the “inflationary epoch” and up to the present era may be interpreted as cosmological synaptic pruning. Even more speculative is to compare the idea of cyclic cosmological models [60, 61, 111] to repeated training cycles of a vast neural network. This could “explain,” for instance, why spacetime appears to be “trained,” in the sense that late layers are more local than early layers under our approach to the horizon problem: previous aeons adjusted the weights. While such analogies have no particular claim to credence, they are impossible to ignore once more modest connections between the two topics have been broached. An environment populated by such ideas might offer further contributions to network design and/or theoretical physics.

Acknowledgements

We thank Jalynn Roberts, Jessica Garriga, Thomas Naugle, Haley Dozier, Joshua Deaton, Lillie Blackmon, Madeline Leboeuf, and Stephanie Dribus for stimulating discussions and technical assistance.