On functions computed on trees

TL;DR

This paper characterizes which functions can be computed on tree-structured hierarchies by deriving necessary and sufficient conditions using differential equations, with implications for neural networks and computational models.

Contribution

It introduces a set of differential equation constraints that determine the functions realizable on tree-structured computation graphs, bridging structure and function.

Findings

Necessary constraints via non-linear PDEs for functions on trees

Sufficient conditions for analytic and bit-valued functions

Enumeration of discrete functions with few realizable cases

Abstract

Any function can be constructed using a hierarchy of simpler functions through compositions. Such a hierarchy can be characterized by a binary rooted tree. Each node of this tree is associated with a function which takes as inputs two numbers from its children and produces one output. Since thinking about functions in terms of computation graphs is getting popular we may want to know which functions can be implemented on a given tree. Here, we describe a set of necessary constraints in the form of a system of non-linear partial differential equations that must be satisfied. Moreover, we prove that these conditions are sufficient in both contexts of analytic and bit-valued functions. In the latter case, we explicitly enumerate discrete functions and observe that there are relatively few. Our point of view allows us to compare different neural network architectures in regard to their function spaces. Our work connects the structure of computation graphs with the functions they can implement and has potential applications to neuroscience and computer science.

Equations

$$F(x_1,\dots ,x_n)=\sum _{i=1}^{2n+1}{f}_i\left(\sum _{j=1}^n{\varphi }_{i,j}(x_j)\right)$$

$$\text{output}=F(x,y,z)=g(f(x,y),z).$$

$$\frac{{\partial }^{2}F}{\partial x\partial z}.\frac{\partial F}{\partial y}=\frac{{\partial }^{2}F}{\partial y\partial z}.\frac{\partial F}{\partial x}.$$

$$F(x,y,z)=xyz+x+y+z.$$

$$F(x,y,z,w)=g(h(f(x,y),z),w)$$

$$F(x,y,z,w)=g(f(x,y),h(z,w))$$

$$\frac{{\partial }^{2}F}{\partial x_i\partial x_l}.\frac{\partial F}{\partial x_j}=\frac{{\partial }^{2}F}{\partial x_j\partial x_l}.\frac{\partial F}{\partial x_i}.$$

$$\det \left[\begin{array}{ccccccc}{F}_x& F_y& F_z& 0& 0& 0& 0\\ F_{xy}& F_{yy}& F_{yz}& F_y& 0& 0& 0\\ F_{xz}& F_{yz}& F_{zz}& 0& F_z& 0& 0\\ F_{xyz}& F_{yyz}& F_{yzz}& F_{yz}& F_{yz}& 0& 0\\ F_{xyy}& F_{yyy}& F_{yyz}& 2F_{yy}& 0& F_y& 0\\ F_{xzz}& F_{yzz}& F_{zzz}& 0& 2F_{zz}& 0& F_z\\ F_{xyzz}& F_{yyzz}& F_{yzzz}& F_{yzz}& 2F_{yzz}& 0& F_{yz}\end{array}\right]=0$$

$$C_i\frac{d{V}_i}{dt}=-\frac{V_i}{R_i}+\sum _{j\text{ is a child of }i}\frac{V_j-V_i}{R_{ij}}-\sum _{c\in \{{\mathrm{Ca}}^{2+},{\mathrm{K}}^{+},{\mathrm{Na}}^{+}\}}{g}_c(V_i)(V_i-V_i^{0,c}).$$

$$\mathbf{F}:(C^{\infty }{)}^n\to C^{\infty }\mathbf{f}=(f_1,\dots ,f_n)\mapsto \mathbf{F}(f_1,\dots ,f_n)$$

$${\mathrm{D}}_{{\mathbf{e}}_i,{\mathbf{e}}_l}^2\mathbf{F}(\mathbf{f}).{\mathrm{D}}_{{\mathbf{e}}_j}\mathbf{F}(\mathbf{f})={\mathrm{D}}_{{\mathbf{e}}_j,{\mathbf{e}}_l}^2\mathbf{F}(\mathbf{f}).{\mathrm{D}}_{{\mathbf{e}}_i}\mathbf{F}(\mathbf{f})$$

$$\begin{array}{rl}& {\mathrm{D}}_{{\mathbf{e}}_s}\mathbf{F}(\mathbf{f})=\underset{h\to 0}{\lim }\frac{\mathbf{F}(\mathbf{f}+h{\mathbf{e}}_s)-\mathbf{F}(\mathbf{f})}{h};\\ & {\mathrm{D}}_{{\mathbf{e}}_s,{\mathbf{e}}_t}^2\mathbf{F}(\mathbf{f})=\underset{h\to 0}{\lim }\frac{\mathbf{F}(\mathbf{f}+h{\mathbf{e}}_s+h{\mathbf{e}}_t)-\mathbf{F}(\mathbf{f}+h{\mathbf{e}}_s)-\mathbf{F}(\mathbf{f}+h{\mathbf{e}}_t)+\mathbf{F}(\mathbf{f})}{h^2}.\end{array}$$

$$\mathcal{F}(T)=\left\{f(F_1,\dots ,F_m)|f\in D_m,F_i\in \mathcal{F}(T_i)\right\}$$

$$D_n=\left\{F:\{0,1{\}}^n\to \{0,1\}\right\}.$$

$$F(x,y,z)=g\left(f(x,y),z\right)$$

$$\frac{\partial F}{\partial x}=\frac{\partial g}{\partial u}.\frac{\partial f}{\partial x},\frac{\partial F}{\partial y}=\frac{\partial g}{\partial u}.\frac{\partial f}{\partial y}$$

$$\frac{{\partial }^{k+1}F}{\partial x\partial z^k}=\frac{{\partial }^{k+1}g}{\partial u\partial z^k}.\frac{\partial f}{\partial x},\frac{{\partial }^{k+1}F}{\partial y\partial z^k}=\frac{{\partial }^{k+1}g}{\partial u\partial z^k}.\frac{\partial f}{\partial y}.$$

$$\frac{{\partial }^{k+1}F}{\partial x\partial z^k}.\frac{\partial F}{\partial y}=\frac{{\partial }^{k+1}F}{\partial y\partial z^k}.\frac{\partial F}{\partial x};$$

$$\frac{{\partial }^{2}F}{\partial x\partial z}.\frac{\partial F}{\partial y}=\frac{{\partial }^{2}F}{\partial y\partial z}.\frac{\partial F}{\partial x}$$

$$\frac{{\partial }^{k+2}F}{\partial x\partial z^{k+1}}.\frac{\partial F}{\partial y}+\frac{{\partial }^{k+1}F}{\partial x\partial z^k}.\frac{{\partial }^{2}F}{\partial y\partial z}=\frac{{\partial }^{k+2}F}{\partial y\partial z^{k+1}}.\frac{\partial F}{\partial x}+\frac{{\partial }^{k+1}F}{\partial y\partial z^k}.\frac{{\partial }^{2}F}{\partial x\partial z}.$$

$$\frac{{\partial }^{k+1}F}{\partial x_i\partial x_l^k}.\frac{\partial F}{\partial x_j}=\frac{{\partial }^{k+1}F}{\partial x_j\partial x_l^k}.\frac{\partial F}{\partial x_i}\text{(for all }k\in \mathbb{N}\text{ and }{x}_i,x_j,x_l\text{ as in Theorem refmain)}.$$

$$F=F(x_1,\dots ,x_n)$$

$$\frac{{\partial }^{1+k_1+\cdots +k_s}F}{\partial x_i\partial x_{l_1}^{k_1}\dots \partial x_{l_s}^{k_s}}.\frac{\partial F}{\partial x_j}=\frac{{\partial }^{1+k_1+\cdots +k_s}F}{\partial x_j\partial x_{l_1}^{k_1}\dots \partial x_{l_s}^{k_s}}.\frac{\partial F}{\partial x_i}$$

$$\frac{{\partial }^{k+1}F}{\partial x_i\partial x_{l_1}^k}.\frac{\partial F}{\partial x_j}=\frac{{\partial }^{k+1}F}{\partial x_j\partial x_{l_1}^k}.\frac{\partial F}{\partial x_i}$$

$$\left(\xi (x,y,z),\eta (x,y,z),z\right)$$

$$\xi (x,y,z):=F(x,y,0),\eta (x,y,z):=y.$$

$$\frac{\partial (\xi ,\eta ,z)}{\partial (x,y,z)}=\left[\begin{array}{ccc}\frac{\partial F}{\partial x}(x,y,0)& \frac{\partial F}{\partial y}(x,y,0)& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$$

$$F(x,y,z)=\sum _{k=0}^{\infty }\frac{1}{k!}\frac{{\partial }^{k}F}{\partial z^k}(x,y,0)z^k;$$

$$\frac{\partial }{\partial \eta }\left(\frac{{\partial }^{k}F}{\partial z^k}(x,y,0)\right)=\frac{{\partial }^{k+1}F}{\partial x\partial z^k}(x,y,0).\frac{\partial x}{\partial \eta }+\frac{{\partial }^{k+1}F}{\partial y\partial z^k}(x,y,0).\frac{\partial y}{\partial \eta }.$$

$$\frac{\partial (x,y,z)}{\partial (\xi ,\eta ,z)}=\left[\begin{array}{ccc}\frac{1}{\frac{\partial F}{\partial x}(x,y,0)}& -\frac{\frac{\partial F}{\partial y}(x,y,0)}{\frac{\partial F}{\partial x}(x,y,0)}& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$$

Full text

1. 1 Introduction

2. 2 Outline and overview of main results

3. 3 Discussion

4. 4 Tree functions

5. 5 Analytic function setting

6. 5.1 The case of ternary functions

7. 5.2 Proof of the main theorem

8. 5.3 Smooth function setting

9. 5.4 Reducing the number of constraints

10. 5.5 Polynomial function setting

11. 6 Bit-valued function setting

12. 6.1 Discrete tree functions

13. 6.2 A metric on the set of labeled binary trees

14. 7 Application to neural networks

15. 7.1 From neural networks to trees

16. 7.2 Trees with repeated labels; a toy example

17. 7.3 Trees with repeated labels; estimates

1. Introduction

A complicated function can be constructed by a hierarchy of simpler functions. For instance, when a microprocessor calculates the value of a function for a given set of inputs, it computes the function through composing simpler implemented functions, e.g. logic gates [HG91, GHR92]. Another example is that of an addition function for any number of inputs which can be obtained by composing simpler addition functions of two inputs. Even when the set of simple functions is small, like in the case of working only with logic gates, the set of functions that can be built may be exponentially large. We know from computation theory that all computable functions can be constructed in this manner [Sip06, AB09]. Therefore, one approach to understand the set of computable functions is to investigate their potential representations as hierarchical compositions of simpler functions.

Here we study the set of functions of multiple variables that can be computed by a hierarchy of functions that each accepts two inputs. Such compositions can be characterized by binary rooted trees (in the following we will refer to them as binary trees) that determines the hierarchical order in which the functions of two variables are applied. Associated with any binary tree is a (continuous or discrete) tree function space (TFS) consisting of all functions that can be obtained as a composition (superposition) based on the hierarchy that the tree provides. In Theorem 2.2 we exhibit a set of necessary and sufficient conditions for analytic functions of $n$ variables to have a representation via a given tree. We show that this amounts to describing the corresponding TFS as the solution set to a group of non-linear partial differential equations (PDEs). We also study the same representability problem in the context of discrete functions.

Related mathematical background. Representing multivariate continuous functions in terms of functions of fewer variables has a rich background that roots back to the $13^{\rm{th}}$ problem on David Hilbert’s famous list of mathematical problems for the $20^{\rm{th}}$ century [Hil02]. Hilbert’s original conjecture was about describing solutions of $7^{\rm{th}}$ degree equations in terms of functions of two variables. The problem has many variants based on the category of functions – e.g. algebraic, analytic, smooth or continuous – in which the question is posed. See [VH67, chap. 1] or the survey article [Vit04] for a historical account. Later in the 1950s, the Soviet mathematicians Andrey Kolmogorov and Vladimir Arnold did a thorough study of this problem in the context of continuous functions that culminated in the seminal Kolmogorov-Arnold Representation Theorem ([Kol57]) which asserts that every continuous function $F(x_{1},\dots,x_{n})$ can be described as

[TABLE]

for suitable continuous single variable functions $f_{i}$, $\phi_{i,j}$.111There are more refined versions of this theorem with more restrictions on the single variable functions that appear in the representation [Lor66, chap. 11]. So in a sense, addition is the only real multivariate function. The idea of applying Kolmogorov-like results to studying networks is not new. Based on the mathematical works of Anatoli Vituškin (see below), the article [GP89] argues that to quantify the complexity of a function, its number of variables (not a suitable indication of the complexity due to Kolmogorov-Arnold Theorem) must be combined with the degree of smoothness due to the fact that there are highly regular functions that cannot be represented by continuously differentiable functions of a smaller number of variables [Vit54]. The paper then concludes that it is not possible to obtain an exact representation usable in the context of network theory because of this emergence of non-differentiable functions. Nevertheless, the article [Ků91] argues that there is an approximation result of this type.

Although pertinent to our discussion, the reader should be aware that the representations of multivariate functions studied in this article are different in following ways:

• •

Motivated by both the structures of computation graphs and the model of neurons as binary trees, we desire multivariate functions that could be obtained via composition of functions of two variables instead of single variable ones.

• •

Unlike the summation above, we work with a single superposition of functions. In the presence of differentiability, this enables us to use the full power of the chain rule. In the case of ternary functions for instance, a typical question (to be addressed in §5.1) would be whether $F(x,y,z)$ can be written as $g\left(f(x,y),z\right)$. In fact, if one allows sum of superpositions, a result of Arnold (which could be found in his collected works [Arn09b]) states that every continuous $F(x,y,z)$ can be written as a sum of nine superpositions of the form $g\left(f(x,y),z\right)$. But we look for a single superposition, not a sum of them.

• •

We mostly work in the analytic context; see §5.3 for difficulties that may arise if one works with smooth functions. It must be mentioned that assuming that the continuous $F(x_{1},\dots,x_{n})$ has certain regularity (e.g. smooth or analytic) does not guarantee that in a representation such as (1.1) the functions can be arranged to be of the same smoothness class [Vit64]. In fact, it is known that there are always $C^{k}$ functions222A function is called (of class) $C^{k}$ if it is differentiable of order $k$ and its $k^{\rm{th}}$ order partial derivatives are continuous. A ($C^{k}$) $C^{1}$ function is said to be (resp. $k$ times) continuously differentiable. A function which is infinitely many times differentiable is called smooth or (of class) $C^{\infty}$. The smaller class of (real) analytic functions that are locally given by convergent power series is denoted by $C^{\omega}$. We refer the reader to [Pug02] for the standard material from elementary mathematical analysis. of three variables which cannot be represented as sums of superpositions of the form $g\left(f(x,y),z\right)$ with $f$ and $g$ being $C^{k}$ as well [Vit54]. Because of the constraints that we put on $F$ in our main result, Theorem 2.2, it turns out that the functions of two variables that appeared in tree representations are analytic as well, or even polynomial if $F$ is polynomial; see Proposition 5.9.

• •

Applying the chain rule to superpositions of analytic (or just $C^{2}$) bivariate functions results in the PDE constraints (2.4). It must be mentioned that the fact that the partial derivatives of functions appearing in any superposition of differentiable functions must be related to each other is by no means new. Hilbert himself has employed this point of view to construct analytic functions of three variables that are not (single or sum of) superpositions of analytic functions of two variables [Arn09a, p. 28]. Ostrowski for instance, has used this idea to exhibit an analytic bivariate function that cannot be represented as a superposition of single variable smooth functions and multivariate algebraic functions due to the fact that it is not a solution to any non-trivial algebraic PDE [Ost20], [Vit04, p. 14]. But, to the best of our knowledge, a systematic characterization of superpositions of analytic bivariate functions as outlined in Theorem 2.2 (or its discrete version in Theorem 6.6) and utilizing that for studying tree functions and neural networks has not appeared in the literature before.

Neuroscience motivation. Over a century ago, the father of modern neuroscience, Santiago Ramón y Cajal, drew the distinctive shapes of neurons [yC95]. Neurons receive their inputs on their dendrites which both exhibit non-linear functions and have a tree structure. The trees, called morphologies, are central to neuron simulations [HC97, Rei99]. Neuronal morphologies are not just the distinctive shapes of neuron but also pertain to their functions. One approximate way of thinking about neural function is that neurons receive inputs and by passing from the dendritic periphery towards the root, the soma, implement computation which gives a neuron its input-output function. In that view, the question of what a neuron with a given dendritic tree and inputs may compute boils down to the question of characterizing its TFS.

The paper is organized as follows. §2 is devoted to a detailed outline the paper. We also state the main results after a non-technical motivation. In §3 we discuss the relevant literature from both computer science and neuroscience sides of the theory. Several possible extensions and open problems are stated as well. The central concept of this paper, tree functions, is formally defined in §4. Sections 5 and 6 treat tree functions in analytic and bit-valued contexts respectively. Finally, in §7 we apply these ideas to study neural networks via tree functions.

2. Outline and overview of main results

The order of appearance of functions in a superposition can be represented by a tree whose leaves, nodes (branch points) and root represent inputs, functions occurring in the superposition and the output respectively. Here we assume that the tree $T$ and the set of functions that could be applied at each node are given and each leaf is labeled by a variable. We can now define the space of functions generated through superposition, i.e. the corresponding tree function space (TFS) (see Definition 4.1). The most tangible case of a TFS is when all of the inputs are real numbers and the functions assigned to the nodes are bivariate real-valued functions. Nonetheless, our definition in §4 covers other cases: an arbitrary tree and sets of functions associated with its nodes result in the set of functions represented by superpositions. One example is when the functions at the nodes are bit-valued functions. Another example is when the inputs are time-dependent and the functions at nodes are operators. The latter case is important since it contains the function that a neural morphology would implement when we only allow soma-directed influences and ignore back-propagating action potentials [SSSH97].

The smallest non-trivial tree representing a superposition is the one with three leaves illustrated in Figure 1(b). Denoting the inputs by $x$, $y$ and $z$, an element $F$ of the corresponding TFS is the superposition below of a function of two variables $f$ and $g$:

[TABLE]

For $f$ and $g$ multiplication or addition, we end up with two basic examples $F(x,y,z)=xyz$ and $F(x,y,z)=x+y+z$. By changing $f$ and $g$ one can construct other examples and hence the question of which functions could be answered in this manner.

To find a necessary condition for a function of three variables $F(x,y,z)$ to have a representation such as (2.1), we assume differentiability and take the derivative. A straightforward application of the chain rule to (2.1) shows that $F$ must satisfy

[TABLE]

A detailed treatment may be found in the discussion from the beginning of §5.1. This partial differential equation for $F$ puts a constraint on functions in the TFS and hence rules out certain ternary functions such as

[TABLE]

While (2.2) is only a necessary condition, we prove that it is also sufficient in the case of analytic ($C^{\omega}$) functions:

Proposition 2.1**.**

Let $F=F(x,y,z)$ be an analytic function defined on an open neighborhood of the origin $\mathbf{0}\in\mathbb{R}^{3}$ that satisfies the identity in (2.2). Then there exist analytic functions $f=f(x,y)$ and $g=g(u,z)$ for which $F(x,y,z)=g\left(f(x,y),z\right)$ over some neighborhood of the origin.

To prove Proposition 2.1, we look at the Taylor expansion of $F$ with respect to $z$ and argue that each partial derivative has a representation like (2.1). We then explicitly construct the desired $f$ and $g$ in (2.1) with the help of the Taylor series. Consequently, we arrive at a description of the TFS containing analytic functions of three variables as the set of solutions to a single PDE.

Generalizing this setup to a higher number of variables, the following question arises: When can an analytic multivariate function be obtained from composition of functions of two variables? Allowing more than three leaves results in graph-theoretically distinct binary trees. For example, in the case of functions of four variables, there exist two non-isomorphic binary trees; Figure 1. The corresponding representations are

[TABLE]

for the first tree and

[TABLE]

for the second one. Thus each (labeled) binary tree $T$ comes with its corresponding space $\mathcal{F}(T)$ of analytic tree functions that could be obtained from analytic functions on smaller number of variables via composition according to the hierarchy that $T$ provides; see Definition 4.1.

Condition (2.2) from the ternary case is the prototype of constraints that general smooth functions from a TFS must satisfy. By fixing $n-3$ variables of a function of $n$ variables in the TFS under consideration, the resulting function of three variables belongs to the TFS of the tree formed by those three leaves and is hence a solution to a PDE of the form (2.2). Since this is true for any triple of variables, numerous necessary conditions must be imposed. In Theorem 2.2 we prove that for analytic functions, these conditions are again sufficient.

Theorem 2.2**.**

Let $T$ be a binary tree with $n$ terminals and $F\in\mathcal{F}(T)$. Suppose the terminals of $T$ are labeled by the coordinate functions $x_{1},\dots,x_{n}$ on $\mathbb{R}^{n}$. Then for any three leaves of $T$ corresponding to variables $x_{i},x_{j},x_{l}$ of $F$ with the property that there is a sub-tree of $T$ containing the leaves $x_{i},x_{j}$ while missing the leaf $x_{l}$ (Figure 2)333Clearly, there is always a rooted sub-tree that separates one of the leaves $x_{i},x_{j},x_{l}$ from the other two: consider the smallest rooted sub-tree (cf. §4 or Figure 6 for the terminology) that has all of them as leaves. Adjacent to the root of this smaller binary tree are its left and right sub-trees. One of them must contain two of $x_{i},x_{j},x_{l}$ and the other one has the third leaf., $F$ must satisfy

[TABLE]

Conversely, an analytic function $F$ defined in a neighborhood of a point $\mathbf{p}\in\mathbb{R}^{n}$ can be implemented on the tree $T$ provided that for any triple $(x_{i},x_{j},x_{l})$ of its variables with the above property (2.4) holds and moreover, for any two sibling leaves $x_{i},x_{i^{\prime}}$, either $\frac{\partial F}{\partial x_{i}}(\mathbf{p})$ or $\frac{\partial F}{\partial x_{i^{\prime}}}(\mathbf{p})$ is non-zero.

The general argument for the trees with larger number of leaves builds on the proof in the case of ternary functions demonstrated above. The proof occupies §5.2 and heavily uses the analyticity assumption. We digress to the setting of smooth ($C^{\infty}$) function in §5.3 to show that this assumption cannot be dropped.

The constraints in the Theorem 2.2 are algebraically dependent. The number of the constraints imposed by Theorem 2.2 is ${n\choose 3}$, hence cubic in the number of leaves. In §5.4 we show that “generically” the number of constraints could be reduced to ${n-1\choose 2}$. This leads to a heuristic444We use the term “heuristic” both because the space of analytic functions that Theorem 2.2 deals with is infinite-dimensional and so one should be careful about the exact meaning of co-dimension; and moreover, because the number of constraints is not necessarily synonymous to the deficit of the dimension of the subspace of tree functions. This is because each constraint in the form of (2.4) is a functional identity that could amount to multiple constraints on the coefficients of the Taylor expansion. result on the co-dimension of the TFS in Proposition 5.6 which states that the number of independent functional equations describing a TFS grows only quadratically with the number of leaves.

The space of analytic functions is infinite-dimensional and this makes it difficult to rigorously measure how “small” a TFS is relative to the ambient space of all analytic functions. However, under certain restrictions, the dimensions of the tree function space or even the space itself are finite. Two examples are worthy of investigation: bit-valued functions of the form $\{0,1\}^{n}\rightarrow\{0,1\}$ and polynomials of bounded degree. In the bit-valued setting of §6.1 each node is characterized by a function $\{0,1\}^{2}\rightarrow\{0,1\}$. We prove that Theorem 2.2 still holds in the sense of formal differentiation; see Theorem 6.6. Moreover, we enumerate the functions in the discrete TFS as $\frac{2\times 6^{n}+8}{5}$ (Corollary 6.2), a number which is much smaller than the number of all possible bit-valued functions which is $2^{2^{n}}$. We use this to conclude that the total number of tree functions of $n$ variables obtained from all labeled binary trees of $n$ leaves is $o\left(2^{2^{n}}\right)$; see Corollary 6.3. In the polynomial setting, each node is a bivariate polynomial. We establish that the Theorem 2.2 applies and furthermore, holds globally; cf. Proposition 5.9. In this case, if we consider polynomials of $n$ variables and of degree not greater than $k$, the polynomial TFS would be of an algebraic variety whose dimension does not exceed $n(k^{2}+1)$; see Proposition 5.11. Again, observe that this is much smaller than the dimension $\binom{k+n}{n}$ of the ambient polynomial space.

The set of binary functions that can be implemented on a given tree is limited and this set allows the reconstruction of the underlying tree from its corresponding TFS; see Proposition 6.9. For two labeled trees we define a metric: the proportion of functions that can only be represented by one of the trees (§6.2). This can be useful: in the case of two neurons with different morphologies, this simple metric quantifies how similar the sets of functions are that the two neurons can implement.

In a more general manner, the functions defined by neural networks are interesting examples of superpositions. In §7.1 we discuss a procedure of “expanding” a neural network to a tree by forming the corresponding Tree Expansion of the Neural Network or TENN for short. The idea is to convert the neural network of interest into a tree by duplicating the nodes that are connected to more than one node of the upper layer; see Figure 4. The procedure then allows us to revert back to the familiar setting of trees. A crucial point to notice is that, unlike previous sections, trees associated with neural networks are not necessarily binary and furthermore, a variable could appear as the label of more than one leaf. In other words, the functions constituting the superposition may have variables in common, e.g. $F(x,y,z)=g(f(x,y),h(x,z))$. Seeking similar constraints for describing the TFS of a tree with repeated labels, in §7.2 we take a closer look at the preceding superposition to obtain a necessary condition:

Proposition 2.3**.**

Assuming that $f,g,h$ are four times differentiable, the superposition $F(x,y,z)=g(f(x,y),h(x,z))$ satisfies the PDE below:

[TABLE]

The proposition suggests that tree functions are again solutions to (perhaps more tedious) PDEs. It is intriguing to ask if in presence of repeated labels there is a characterization, similar to Theorem 2.2, of a TFS as the solution set to a system of PDEs; cf. Question 7.2. We finish with one final application of this idea of transforming a neural network to a tree: In Theorem 7.9 of §7.3, we give an upper bound on the number of bit-valued functions computable by a neural network in terms of parameters depending on the architecture of the network.

3. Discussion

Here we study the functions that are obtained from hierarchical superpositions; we study functions that can be computed on trees. The hierarchy is represented by a rooted binary tree where the leaves take different inputs and at each node a bivariate function is applied to outputs from the previous layer. In the setting of analytic functions, in Theorem 2.2 we characterize the space of functions that could be generated accordingly as the solution set to a system of PDEs. This characterization enables us to construct examples (e.g. (2.3)) of functions that could not be implemented on a prescribed tree. This is reminiscent of Minsky’s famous XOR theorem [MP17]. The space of analytic functions is infinite-dimensional and this motivates us to investigate two settings in which the TFS is finite-dimensional (polynomials) or even finite (bit-valued). We show that the dimension or size of the TFS is considerably smaller than that of the ambient function space. The number of bit-valued functions could be estimated even for non-binary trees following the same ideas. Finally, we bridge between trees and neural networks by associating with each feed-forward neural network its corresponding TENN; cf. Figure 4. This procedure allows us to apply our analysis of trees yielding an upper bound for the number of bit-valued functions that can be implemented by a neural network.

Our main result in the continuous setting, Theorem 2.2, holds only for analytic functions; see the discussion in §5.3. While they constitute a large class of functions, there are important cases where one must deal with continuous non-analytic functions too. For example, a typical deep learning network is built through composition from analytic functions such as linear and sigmoid functions or the hyperbolic tangent; and also, from non-analytic functions such as ReLU or the pooling function. Continuous functions could be approximated locally by analytic ones with any desired precision (although not over the entirety of the real axis). Therefore, while our main result (local by formulation) is an exact classification, one future direction is to study how well arbitrary continuous functions could be approximated by analytic tree functions.

The morphology of the dendrites of neurons processes information through a (commonly binary [KD05, GA15]) tree and so it is naturally related to the setup of this paper; see Figure 5 [Mel94, PBM03]. A typical dendritic tree receives synaptic inputs from surrounding neurons. When activated, the synapses induce a current which changes the voltage on the dendrite. This is followed by a flow of the resulting current towards (and away from) the root (soma) of the neuron. In typical models of neural physiology, a neuron is segmented into compartments where their connections and the biological parameters define the dynamics of the voltage for each compartment [Seg98]. The dynamics of the electrical activity is often given by the following well-known ODE555The quantities appeared here are:

•

$V_{i}$ is the voltage potential of the $i^{\rm{th}}$ compartment;

•

$V_{i}^{0,c}$ is the resting voltage potential for the $i^{\rm{th}}$ compartment and the ion $c$;

•

$C_{i}$ is the membrane capacitance of the $i^{\rm{th}}$ compartment;

•

$R_{i},R_{i,j}$ denotes the resistance of the $i^{\rm{th}}$ compartment and $R_{i,j}$ is the resistance between $i^{\rm{th}}$ and $j^{\rm{th}}$ compartments;

•

$g_{c}$ is non-linear function corresponding to the ion $c$.

We only consider currents towards the soma. [HC97]:

[TABLE]

Consequently, in the case of time-varying inputs, TFSs could be of neuroscientific interest. In this situation, the functions at the nodes are operators that receive time-dependent functions as their inputs. Constraints such as (2.4) in the main theorem may be formulated in this case as well: An operator

[TABLE]

admitting a tree representation is expected to satisfy equations such as

[TABLE]

where derivatives of the operator must be understood in the variational sense

[TABLE]

Utilizing discrete TFSs, in §6.2 we introduce a metric on the set of labeled binary trees that may be potentially used to quantify how similar two neurons are. A careful adaptation of our results to the time-varying situation could be the object of future enquiries.

Certain assumptions must be made before any application of our treatment of tree functions to the study of neural morphologies. First, from a biological standpoint not all functions are admissible as functions applied at nodes of neurons. Secondly, the acyclic nature of trees assumes that a neuron functions only due to feed-forward propagation whereas in reality back-propagating action potentials also occur. Thirdly, it is well-known that there are biological mechanisms, such as ephaptic connectivity or neuromodulations, that could affect the computations in a neuron’s morphology, and they are not taken into account in typical compartmental models. Our approach only applies to an abstraction of models; however, this abstraction appears meaningful.

In this paper, the “complexity” of a TFS in bit-valued and polynomial settings is measured by its cardinality or dimension. However, there are other notions of complexity in the literature that try to capture the capacity of the space of computable functions. For example, the VC-dimension measures the expressive power of a space of functions by quantifying the set of separable stimuli [VLC94, ZP96, MLP17]. For linear combinations of smooth kernels, the minimum number of bases can measure the complexity of a classifier [BDLR05, BDR06]. This suggests that large complexities of shallow networks might be due to the “amount of variations” of functions to be computed [KS16]. When the functions at the nodes are piece-wise linear, one can count the number of linear regions of the output function [MPCB14, HR19]. The choice of the complexity measurement method is important when it comes to quantifying the difference between two architectures.

When a model is trained on data, we search for the best fit in the function space. Characterizing the landscape of function space can present new methods for training [BRK19]. To train models in machine learning, we use a variety of methods such as (stochastic) gradient decent, genetic algorithms, or more recent methods such as learning by coincidence [SL18]. For some models such as regression, we have explicit formulae that show how to find the parameters from training data. One future line of research is to investigate whether our PDE description of the TFSs can point toward new methods of training.

Since a TFS is much smaller than the ambient space of functions, it is suggestive to consider the approximation by them. In this regard, we fix a target function and take into account the tree functions that approximate it. Searching for the best approximation of a target function in the function space is realized by the training process. Hence one important question for approximation of a function is the stability of this process [HR17]. Another approach is to develop the mean-field equations to approximate the function space with a fewer equations that are easier to handle [MMN18]. Poggio et al. have found a bound for the complexity of a neural networks with smooth (e.g. sigmoid) or non-smooth (e.g. ReLU) non-linearity that provides a prescribed accuracy for functions of a certain regularity [PMR*+*17]. Also by estimating the statistical risk of a neural network, one can describe model complexity relative to sample size [BK18]. Now that we have a description of analytic tree functions as solutions to a system of PDEs, one further direction is to study approximations of arbitrary continuous functions by these solutions.

When the tree function is fed the same input more than once through different leaves, the constraints put on superpositions in Theorem 2.2 must be refined and become more tedious. In §7.2, we study the simplest possible case, namely the superposition (7.1). Computing higher derivatives via the chain rule along with a linear algebra argument yield the complicated fourth order PDE of Proposition 2.3 as a constraint. One future line of research is to formulate similar PDE constraints in the case of general (not necessary binary) trees with repeated labels; cf. Question 7.2. Finding necessary or sufficient constraints in the repeated regime would have immediate applications to the study of continuous functions computed via neural networks with this consideration in mind that for the TENN associated with a neural network even functions assigned to nodes are probably repeated. Moreover, in the more specific context of polynomial functions, it is promising to try to formulate results such as Proposition 5.11 about the space of polynomial tree functions; or in the bit-valued setting, any strengthening of the bound on the number of bit-valued functions implemented on a general tree that Corollary 7.7 provides would be desirable.

One major goal of the theoretical deep learning is to understand the role of various architectures of neural networks. Previous studies have shown that, compared to shallow networks, deep networks can represent more complex functions [BS14, LTR17, KS17]. Comparing VC-dimension of different architectures is insightful into why high-dimensional deep networks trained on large training sets often do not seem to show overfit [MLP17]. Theorem 7.9 from the last section of this paper provides further intuition in this direction once instead of more traditional fully connected multi-layer perceptrons, we work with currently more popular sparse neural networks (e.g. convolutional neural networks). This is due to the fact that in the tree expansion of a sparse network the number of children of any arbitrary node would be relatively small. Theorem 7.9 indicates that the number of bit-valued functions computable by the network could be large only if the associated tree has numerous leaves. Since the tree is sparse, this could happen only if the depth of the tree (or equivalently, that of the network) is relatively large. The discussion in §7 suggests that studying tree functions could serve as a foundation for interesting theoretical approaches to the study of neural networks.

4. Tree functions

In this section we define the function space associated with a tree in the most general setting. Suppose we have $n$ inputs (leaves) of a binary tree $T$. We recursively compute the output by applying at each node a function and passing the result to the next level. These calculations continue until we reach the root.

Definition 4.1**.**

Let $T$ be a tree and $I$ the set of all possible inputs that a leaf could receive. For any $n\in\mathbb{N}$, suppose $D_{n}\subseteq\{f|f:I^{n}\rightarrow I\}$. The tree function space, $\mathcal{F}(T)$, is defined recursively: $\mathcal{F}(T)=D_{1}$ if $T$ has only one vertex. For larger trees, assuming that the successors of the root of $T$ are the roots of smaller sub-trees $T_{1},\dots,T_{m}$, define:

[TABLE]

Tree function spaces could be investigated in two different regimes:

1. (1)

Functions are real analytic, i.e. $D_{n}=C^{\omega}(U)$ with $U$ an open subset of $\mathbb{R}^{n}$ (§5); 2. (2)

Functions are bit-valued (§6), i.e. they belong to

[TABLE]

Definition 4.2**.**

A tree is called binary if every non-terminal vertex (every node) of it has precisely two successors; cf. Figure 6.

The terminology of binary trees.

• •

Root: the unique vertex with no predecessor/parent.

• •

Leaf/Terminal: a vertex with no successor666The reader is cautioned that in our usage of terms such as “children”, “parent”, “successor” and “predecessor” in reference to vertices we have a rooted tree as illustrated in Figure 6 in mind where the root precedes every other vertex whereas to implement a function, the computations are done in the “upward” direction starting from the leaves in the lowest level and culminating at the root./child.

• •

Node/Branch point: a vertex which is not a leaf, i.e. has (two) successors.

• •

Sub-tree: all descendants of a vertex along with the vertex itself.

• •

Sibling leaves: two leaves with the same parent.

• •

Outsider of a triple of leaves: the leaf in the triple which is not a descendant of any common ancestors of the other two. Equivalently, the one which is separated from the other two leaves via a rooted sub-tree; see Figure 2. For example, in Figure 3, $z$ is the outsider of the triple $\{x,y,z\}$.

Convention. All trees are assumed to be rooted. The number of leaves (terminals) of a tree is always denoted by $n$, and each leaf presents a variable. Unless stated otherwise, the tree is binary and these variables are assumed to be distinct, and hence the corresponding functions are of $n$ variables. Repeated labels come up only in §7.

5. Analytic function setting

5.1. The case of ternary functions

In this section, we focus on the first interesting case, namely a binary tree with three inputs. It turns out that the treatment of this basic case and the ideas therein are essential to the proof of Theorem 2.2. In order to make one output, two of the inputs should be first combined at one node, and the result of that combination is then combined with the third input at the root. Such functions can be written as:

[TABLE]

where $f=f(x,y)$ and $g=g(u,z)$ are two smooth functions of two variables.

So which functions of three inputs, $F$ could be written as in (5.1)? Taking the derivative w.r.t. $x$ and $y$, we have:

[TABLE]

Taking the $k^{\rm{th}}$ derivative w.r.t. $z$ yields:

[TABLE]

Hence for every $k\in\mathbb{N}$ we should have:

[TABLE]

as both sides coincide with $\frac{\partial^{k+1}g}{\partial u\partial z^{k}}\frac{\partial g}{\partial u}\frac{\partial f}{\partial x}\frac{\partial f}{\partial y}$. In particular, for $k=1$

[TABLE]

which is the constraint (2.2) from §2. Notice that the identity is solely based on the function $F$ and serves as a necessary condition for the existence of a presentation such as (5.1) for $F$.

It is essential to observe that constraint (2.2) implies the rest of the constraints imposed on $F$ in (5.2). This is trivial for the points where $\left[\frac{\partial F}{\partial x},\frac{\partial F}{\partial y}\right]=\mathbf{0}$. Otherwise, either $\frac{\partial F}{\partial x}$ or $\frac{\partial F}{\partial y}$ should be non-zero at the point under consideration and hence throughout a small enough neighborhood of it. We proceed by induction on $k$. Differentiating (5.2) w.r.t. $z$ yields:

[TABLE]

We claim that the latter terms of two sides coincide and this will finish the inductive step. From the induction hypothesis $\frac{\partial^{k+1}F}{\partial x\partial z^{k}}.\frac{\partial F}{\partial y}=\frac{\partial^{k+1}F}{\partial y\partial z^{k}}.\frac{\partial F}{\partial x}$, while the base case $k=1$ indicates $\frac{\partial^{2}F}{\partial x\partial z}\frac{\partial F}{\partial y}=\frac{\partial^{2}F}{\partial y\partial z}\frac{\partial F}{\partial x}$. The vectors $\left[\frac{\partial^{k+1}F}{\partial x\partial z^{k}},\frac{\partial^{k+1}F}{\partial y\partial z^{k}}\right]$ and $\left[\frac{\partial^{2}F}{\partial x\partial z},\frac{\partial^{2}F}{\partial y\partial z}\right]$ are multiples of the non-zero vector $\left[\frac{\partial F}{\partial x},\frac{\partial F}{\partial y}\right]\neq\mathbf{0}$; so they are multiples of each other, i.e. $\frac{\partial^{k+1}F}{\partial x\partial z^{k}}.\frac{\partial^{2}F}{\partial y\partial z}=\frac{\partial^{k+1}F}{\partial y\partial z^{k}}.\frac{\partial^{2}F}{\partial x\partial z}$.

In the same vein, identity (2.4) implies the more general identity below:

[TABLE]

This is true even for a greater number of variables $x_{l_{1}},\dots,x_{l_{s}}$ in place of $x_{l}$ in the following sense:

Lemma 5.1**.**

Let $T$ be the tree from Theorem 2.2 and

[TABLE]

be a function of $n$ variables satisfying the constraints (2.4). Then

[TABLE]

provided that for each leaf $x_{l_{t}}$ there is a rooted sub-tree containing $x_{i},x_{j}$ that separates them from $x_{l_{t}}$ (e.g. Figure 2 where $x_{i},x_{j}$ are separated from $x_{l},x_{l^{\prime}}$ through their predecessor).

Proof.

This can be inferred by employing the same inductive argument; i.e. differentiating the identity

[TABLE]

similar to (5.3) w.r.t. variables $x_{l_{2}},\dots,x_{l_{s}}$ and using (2.4) for triples of variables $(x_{i},x_{j},x_{l_{i}})$ along with the non-vanishing of one of $\frac{\partial F}{\partial x_{i}}$ or $\frac{\partial F}{\partial x_{j}}$. One trivially gets equality at points where they vanish simultaneously. ∎

We next argue that locally, the aforementioned condition is sufficient. In another words, if an analytic ternary function satisfies (2.2), then it locally admits a tree representation such as (5.1).

Proof of Proposition 2.1.

Let us first impose a mild non-singularity condition at the origin: either $\frac{\partial F}{\partial x}(\mathbf{0})$ or $\frac{\partial F}{\partial y}(\mathbf{0})$ is non-zero. Without any loss of generality, we may assume $F(\mathbf{0})=0$ and $\frac{\partial F}{\partial x}(\mathbf{0})\neq 0$. The idea is to come up with a new coordinate system

[TABLE]

centered at the origin in which the function $F$ is dependent on only $\xi,\eta$. Define

[TABLE]

The Jacobian of $(\xi,\eta,z)$ w.r.t. $(x,y,z)$ is given by

[TABLE]

whose determinant at the origin is $\frac{\partial F}{\partial x}(\mathbf{0})$ which we have assumed to be non-zero. Thus $(\xi,\eta,z)$ is indeed a coordinate system centered at $\mathbf{0}$. Next, we consider the Taylor expansion of $F(x,y,z)$ w.r.t. $z$:

[TABLE]

the equality which holds near the origin due to the analyticity assumption. We claim that in the new coordinate system $(\xi,\eta,z)$ the partial derivatives $\frac{\partial^{k}F}{\partial z^{k}}(x,y,0)$ that appeared above are independent of $\eta,z$. The latter is clear and for the former we apply the chain rule to differentiate with respect to $\eta$:

[TABLE]

To calculate $\frac{\partial x}{\partial\eta},\frac{\partial y}{\partial\eta}$ one has to invert the Jacobian matrix (5.6):

[TABLE]

that yields $\frac{\partial x}{\partial\eta}(x,y,z)=-\frac{\frac{\partial F}{\partial y}(x,y,0)}{\frac{\partial F(x,y,0)}{\partial x}},\frac{\partial y}{\partial\eta}(x,y,z)=1$. Plugging in the previous expression for $\frac{\partial}{\partial\eta}\left(\frac{\partial^{n}F}{\partial z^{n}}(x,y,0)\right)$ we get:

[TABLE]

which is zero due to (5.2); keep in mind that in a neighborhood of the origin the aforementioned identities are implied by (2.2); cf. Lemma 5.1. We conclude that in (5.7) each term $\frac{\partial^{k}F}{\partial z^{k}}(x,y,0)$ is a function of $\xi(x,y,z)=F(x,y,0)$, e.g.

[TABLE]

Now defining $f(x,y)$ to be $F(x,y,0)$ and $g(u,z)$ to be $\sum_{k=0}^{\infty}\frac{1}{k!}g_{k}(u)z^{k}$, the identity (5.7) implies that $F(x,y,z)=g\left(f(x,y),z\right)$ throughout a small enough neighborhood of $\mathbf{0}\in\mathbb{R}^{3}$.

Next, we omit the assumption that one of the partial derivatives of $F$ is non-zero in Proposition 2.1. If either of $\frac{\partial^{k+1}F}{\partial x\partial z^{k}}(\mathbf{0})$ or $\frac{\partial^{k+1}F}{\partial y\partial z^{k}}(\mathbf{0})$ is non-zero for some integer $k$, we apply what we just proved to $\frac{\partial^{k}F}{\partial z^{k}}$ to get:

[TABLE]

Then integrating $k$ times w.r.t. $z$ provides us with a similar expression for $F$. There is nothing to prove if all of the partial derivatives $\frac{\partial^{k+1}F}{\partial x\partial z^{k}}(\mathbf{0})$ and $\frac{\partial^{k+1}F}{\partial y\partial z^{k}}(\mathbf{0})$ are zero since in that case the Taylor expansion of $F$ describes it as the sum of a function of $(x,y)$ and a function of $z$. ∎

Remark 5.2**.**

The idea from the last part of the proof seems to work only for this particular presentation as in general, integration w.r.t. to one of the variables does not preserve forms such as $g\left(f(x,y),h(z,w)\right)$. Therefore, we are going to need the non-singularity condition of Theorem 2.2 in the following section.

Remark 5.3**.**

An elegant reformulation (from the field of integrable systems) of constraint (2.2) imposed on a smooth tree function $F(x,y,z)$ is to say that the differential form777The reader may find a very readable account of the theory of differential forms on Euclidean spaces in [Pug02, chap. 9, sec. 5]. $\omega:=\frac{\partial F}{\partial x}{\rm{d}}x+\frac{\partial F}{\partial y}{\rm{d}}y$ must satisfy $\omega\wedge{\rm{d}}\omega=0$:

[TABLE]

Similar identities also hold in the general case of a (smooth) tree function $F(x_{1},\dots,x_{n})\in\mathcal{F}(T)$ as has appeared in Theorem 2.2. To any two sibling leaves $x_{i},x_{j}$ assign the differential $1$-form $\omega_{i,j}:=\frac{\partial F}{\partial x_{i}}{\rm{d}}x_{i}+\frac{\partial F}{\partial x_{j}}{\rm{d}}x_{j}$. A straightforward calculation yields $\omega_{i,j}\wedge{\rm{d}}\omega_{i,j}$ as

[TABLE]

which turns out to be zero since any other leaf $x_{l}$ is an outsider with respect to neighboring $x_{i},x_{j}$; hence the terms inside parentheses vanish due to (2.4). The non-vanishing condition of Theorem 2.2 implies that these $1$-forms are linearly independent throughout some small enough open subset of $\mathbb{R}^{n}$. They define a differential system on the aforementioned open subset whose rank is:

[TABLE]

and the identities $\omega_{i,j}\wedge{\rm{d}}\omega_{i,j}=0$ could be reinterpreted as the integrability of this system according to a classical theorem of Frobenius [Nar68, Theorem 2.11.11].

The discussion in this subsection settles Theorem 2.2 for the most basic case of a binary tree with three leaves.

5.2. Proof of the main theorem

Let $T$ be a binary tree with $n$ leaves as in Theorem 2.2 and $F$ be a differentiable function of $n$ variables on an open neighborhood $U$ of $\mathbf{p}\in\mathbb{R}^{n}$.

The proof of necessity

Let $F\in\mathcal{F}(T)$. Consider a triple of variables $(x_{i},x_{j},x_{l})$ as in Theorem 2.2. For the ease of notation, suppose they are the first three coordinates $x_{1},x_{2},x_{3}$. Given $k\in\mathbb{N}$ and a point

[TABLE]

we need to verify (2.4) at $\mathbf{q}$. Setting the last $n-3$ coordinates to be constants $q_{4},\dots,q_{n}\,,$ we end up with the function

[TABLE]

of three variables defined on the open neighborhood $\pi(U)$ of $(q_{1},q_{2},q_{3})\in\mathbb{R}^{3}$ which is the image of $U$ under the projection onto the first three coordinates $\pi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{3}$. This new function is implemented on a tree with three inputs $x,y,z$ (corresponding to leaves $x_{i},x_{j},x_{l}$ in the original statement of Theorem 2.2) and with $x,y$ adjacent to the same node as $x_{i}$ and $x_{j}$ were separated from $x_{l}$ in the original tree; see Figure 3. Hence (2.2) holds for this function:

[TABLE]

which at $(x,y,z)=(q_{1},q_{2},q_{3})$ yields the desired constraint

[TABLE]

Next, we argue that under the assumptions outlined in the second part of Theorem 2.2 the identities such as (2.4) are enough to implement $F(x_{1},\dots,x_{n})$ on $T$ locally around $\mathbf{p}$. The proof of sufficiency is based on recursively constructing the desired presentation of $F$ as a composition of bivariate functions by reducing the size of $T$. The base of the induction, the case of a tree with three terminals, has already been settled in Proposition 2.1.

We claim that, up to relabeling variables, $F(x_{1},\dots,x_{n})$ can be written as either

[TABLE]

or

[TABLE]

where the function $f$ satisfies the hypothesis of the existence part of Theorem 2.2 for $n-1$ or $s$ variables (the integer $s\in\{2,\dots,n-2\}$ is going to be introduced shortly). In terms of the tree $T$, the first normal form occurs when $x_{n}$ is connected directly to the root; the removal of the leaf and the root then results in a smaller tree $T^{\prime}$ with $n-1$ leaves; cf. part (a) of Figure 7. The induction hypothesis then establishes $f\in\mathcal{F}(T^{\prime})$ and finishes the proof. On the other hand, (5.10) comes up when neither of the two rooted sub-trees obtained from excluding the root is singleton. The number $s\geq 2$ here denotes the number of the leaves of one of these sub-trees, e.g. the “left” one. By symmetry, let us assume that variables are labeled such that $x_{1},\dots,x_{s}$ are the leaves of the sub-tree to the left of the root while $x_{s+1},\dots,x_{n}$ appear in the sub-tree to the right. Graph-theoretically, gathering the variables $x_{1},\dots,x_{s}$ together in (5.10) amounts to collapse the left sub-tree to a leaf. This results in a new binary tree $T^{\prime\prime}$ with the same root but with $n-s+1$ leaves which is of the form discussed before: it has a “top” leaf directly connected to the root; see part (b) of Figure 7. The final step is to invoke the induction hypothesis to argue that $g$ in (5.10) belongs to $\mathcal{F}(T^{\prime\prime})$.

Part I of the proof of sufficiency: Suppose there is a leaf adjacent to the root.

Without loss of generality, we consider everything in a neighborhood of $\mathbf{0}\in\mathbb{R}^{n}$ and assume $F(\mathbf{0})=0$. Theorem 2.2 also requires at least one of the partial derivatives of $F$ w.r.t. $x_{1},\dots,x_{n-1}$ to be non-zero; by symmetry, let us assume $\frac{\partial F}{\partial x_{1}}(\mathbf{0})\neq 0$.

Next, define

[TABLE]

This is a new coordinate system centered at the origin as the Jacobian

[TABLE]

is of determinant $\frac{\partial F}{\partial x_{1}}(\mathbf{0})\neq 0$ at the origin. The goal is to write down $F(x_{1},\dots,x_{n})$ in a form

[TABLE]

similar to (5.9) for a suitable bivariate function $g$ and then applying the induction hypothesis to $F(x_{1},\dots,x_{n-1},0)$ which of course satisfies (2.4) with the original tree $T$ replaced with $T^{\prime}$. To this end, we consider the Taylor expansion of $F(x_{1},x_{2},\dots,x_{n})$ w.r.t. $x_{n}$:

[TABLE]

We claim that the functions

[TABLE]

appeared as coefficients are dependent only on the first component of the new coordinate system (5.11), or in other words

[TABLE]

This is immediate when $i=n$ as we are basically differentiating w.r.t. $x_{n}$. For $2\leq i<n$, we need to apply the chain rule to get

[TABLE]

where the partial derivatives $\frac{\partial x_{s}}{\partial y_{i}}$ are entries of the inverse of (5.12) given by

[TABLE]

Hence $\frac{\partial x_{s}}{\partial y_{i}}$ is non-zero only for $s=1,i$ and

[TABLE]

which is zero as (5.3) holds (keep in mind that the sub-tree $T^{\prime}$ of $T$ has $x_{1},x_{i}$ while it misses $x_{n}$ since, as part (a) of Figure 7 demonstrates, the leaf corresponding to $x_{n}$ is connected directly to the root of $T$.). Consequently, the term $\frac{\partial^{k}F}{\partial x_{n}^{k}}(x_{1},\dots,x_{n-1},0)$ in (5.14) can be written as a function $h_{k}(y_{1})$ where $y_{1}$ has been defined to be $F(x_{1},\dots,x_{n-1},0)$ in (5.11). Hence

[TABLE]

works in (5.13). ∎

Part II of the proof of sufficiency: Suppose there is no leaf adjacent to the root.

We work with the convention discussed above: Among the two rooted sub-trees resulting from excluding the root of $T$, the “left” one has variables $x_{1},\dots,x_{s}$ as its leaves while the “right” one has the rest of the variables $x_{s+1},\dots,x_{n}$. The number $s$ is assumed to be larger than $1$ as the case of $s=1$ has just been treated in Part I of the proof of sufficiency.

Expand $F(x_{1},\dots,x_{s};x_{s+1},\dots,x_{n})$ w.r.t. the last $n-s$ variables:

[TABLE]

The non-vanishing assumption of Theorem 2.2 requires the partial derivative at $\mathbf{p}=\mathbf{0}$ of $F$ with respect to at least one of the variables in the left sub-tree to be non-zero; let us assume $\frac{\partial F}{\partial x_{1}}(\mathbf{0})\neq 0$. By a similar Jacobian determinant computation appeared in part I of the proof, the assignment

[TABLE]

defines a coordinate system centered at the origin of $\mathbb{R}^{s}$ as $\frac{\partial F}{\partial x_{1}}(\mathbf{0})\neq 0$. Repeating the argument that has come up multiple times before, the term

[TABLE]

appeared in (5.16) is a function of $\xi(x_{1},\dots,x_{s})$ since for every $2\leq i\leq s$ its derivative w.r.t. the component $x_{i}$ of the system (5.17) is zero due to

[TABLE]

the identity that follows from Lemma 5.1 because the right sub-tree separates $x_{s+1},\dots,x_{n}$ from the leaves $x_{1},x_{i}$ of the left sub-tree. Therefore, (5.16) can be rewritten as

[TABLE]

where the single variable function $u_{k_{s+1},\dots,k_{n}}(\xi)$ satisfies

[TABLE]

The goal is to show that the function

[TABLE]

of $n-s+1$ variables can be represented by the tree $T^{\prime\prime}$ as in that case

[TABLE]

is given by

[TABLE]

which is in the form of (5.10) with $\xi(x_{1},\dots,x_{s})=F(x_{1},\dots,x_{s};\overbrace{0,\dots,0}^{n-s\text{ times}})$ obviously satisfying the induction hypothesis for the sub-tree to the left of the root and $g$ satisfying the same but for the tree $T^{\prime\prime}$ with $n-s+1$ terminals obtained from collapsing the aforementioned left sub-tree of $T$ to a point. To verify the conditions of the theorem for $g$, first observe that for $i,j,l>s$ the identity

[TABLE]

holds trivially. This is due to the fact that by definition

[TABLE]

which yields

[TABLE]

for any $s+1\leq t\leq n$ and besides, $F$ satisfies the analogous constraint

[TABLE]

It needs to be mentioned that furthermore, the non-vanishing requirement of the induction hypothesis can be deduced from (5.22) since

[TABLE]

for every $i>s$; and any leaf $x_{i}$ of the new tree $T^{\prime\prime}$ having a sibling leaf $x_{i^{\prime}}$ has to come from the original tree $T$; hence the desired

[TABLE]

is the same as the non-vanishing condition

[TABLE]

that the theorem has imposed on $F$. Consequently, the challenging part would be to verify

[TABLE]

for any $s<i<j\leq n$. Differentiating (5.21) along with the definition of $\xi(x_{1},\dots,x_{s})$ as $F(x_{1},\dots,x_{s};\overbrace{0,\dots,0}^{n-s\text{ times}})$ yield

[TABLE]

In particular, near the origin where $\frac{\partial F}{\partial x_{1}}\neq 0$, one has

[TABLE]

Taking the partial derivatives of (5.24) w.r.t. variables $x_{i},x_{j}$ (where $i,j>s$) and invoking (5.22), we see that (5.23) amounts to

[TABLE]

which holds since

[TABLE]

that is, one of the constraints imposed on $F$ in (2.4) ($x_{1}$ is separated from $x_{i},x_{j}$ via the right sub-tree). ∎

5.3. Smooth function setting

The proof above heavily relied on the analyticity of the function $F$ under consideration. As a matter of fact, in the context of $C^{\infty}$ functions one needs to strengthen the constraint (2.4) as follows (for simplicity, we have replaced $x_{i}$, $x_{j}$ and $x_{l}$ with $x_{1}$,$x_{2}$ and $x_{3}$ respectively):

[TABLE]

for any two points $(a,b,c;q_{4},\dots,q_{n})$ and $(a,b,c^{\prime};q_{4},\dots,q_{n})$ lying in an open ball on which $F$ is defined. This is best demonstrated for the superposition

[TABLE]

from (5.1): The computations carried out there give us

[TABLE]

which readily implies

[TABLE]

Notice that the stronger constraint (5.25) could be derived from the ordinary ones (2.4) and (5.3) if the function is analytic: Form the single variable function

[TABLE]

defined on an open interval of $z$-values containing both $c$ and $c^{\prime}$. The function vanishes at $z=c^{\prime}$ and we wish to show that it is identically zero as then it would be zero for $z=c$ too. Due to analyticity, it suffices to show that the derivatives of all orders vanish at $z=c^{\prime}$. Notice that the $k^{\rm{th}}$ derivative at that point is

[TABLE]

which is zero due to (5.3). We are going to continue to work in the analytic category hereafter and so there would be no need to generalize constraint (2.4) to identities such as (5.25).

Example 5.4**.**

A typical example of a non-analytic smooth function is

[TABLE]

We are going to use it to construct a smooth (but of course non-analytic) function of three variables $F(x,y,z)$ that satisfies constraint (2.2) but not the generalized one introduced above. Set

[TABLE]

We have

[TABLE]

while

[TABLE]

which coincide since they are both identically zero due to the fact that $\rho$, and therefore $\rho^{\prime}$, vanishes at non-positive numbers. Notice that the generalized condition is not satisfied here:

[TABLE]

is positive when $z>0$ whereas

[TABLE]

is zero in that case. It should not be surprising that in the absence of analyticity, $F(x,y,z)$ serves as a counter example to Proposition 2.1: We are going to argue that there is no representation

[TABLE]

of $F$ in a neighborhood of the origin with $g,h$ continuous functions of two variables. Aiming for a contradiction, suppose there are continuous functions $f,g$ satisfying

[TABLE]

for $(x,y,z)\in[-\epsilon,\epsilon]^{3}$ where $\epsilon>0$. Plugging $z=\epsilon$ and $z=-\epsilon$, we arrive at:

[TABLE]

In conjunction, these two identities imply that $f:[-\epsilon,\epsilon]^{2}\rightarrow\mathbb{R}$ is injective. This is absurd as for obvious topological reasons, there is no continuous injective map from any non-degenerate square to the real line.

5.4. Reducing the number of constraints

Theorem 2.2 provides necessary and sufficient conditions for a multivariate function to belong to the function space associated with a tree. Here, we approach the problem of finding the co-dimension of this infinite-dimensional subspace by counting the number of independent constraints. In general, the condition (2.2) of Theorem 2.2 should hold for any triple of variables and therefore, ${n\choose 3}$ equations for a tree with $n$ leaves. However, many of these equations are redundant. To find the number of algebraically independent equations we shall need the following lemma.

Lemma 5.5**.**

Let $T$ be a binary tree. Denote its left and right sub-trees by $T_{1},T_{2}$ and suppose that they have $s$ and $n-s$ leaves. Then the number of algebraically independent equations corresponding to triples that have elements from both $T_{1}$ and $T_{2}$ is $s(n-s)-1$.

Proof.

Denote the leaves of $T_{1}$ by $x_{1},\dots,x_{s}$ and those of $T_{2}$ by $x_{s+1},\dots,x_{n}$. For the ease of notation, we switch to the subscript notation for partial derivatives. If $x_{i}$ is another leaf of $T_{1}$ (different from $x_{1}$) and $x_{j}$ is another leaf for $T_{2}$ (different from $x_{n}$), then:

[TABLE]

Here, we have used the constraint (2.4) for triples $x_{1},x_{i},x_{j}$ and $x_{1},x_{j},x_{n}$ where in the former (resp. the latter) two of the variables are in the sub-tree $T_{1}$ (resp. $T_{2}$) while the third one is in the other sub-tree. As $(x_{i},x_{j})$ varies among all the pairs formed by the leaves $x_{i}$ of $T_{1}$ and $x_{j}$ of $T_{2}$, we get $s(n-s)-1$ constraints as we need $s(n-s)$ fractions of the form $\frac{F_{x_{i}x_{j}}}{F_{x_{i}}F_{x_{j}}}$ to coincide. ∎

Proposition 5.6**.**

There are ${n-1\choose 2}$ algebraically independent constraints for a tree $T$ with $n$ leaves.

Proof.

This is clear for the first few values of $n$ as the number of constraints is zero when $n=1,2$ and is $\binom{3}{3}=1$ for $n=3$. Fix a tree $T$ with $n\geq 3$ leaves and denote its left and right sub-trees by $T_{1},T_{2}$. Suppose they have $s,n-s$ leaves respectively. By induction we know that there are ${s-1\choose 2}$ and ${n-s-1\choose 2}$ algebraically independent equations corresponding to sub-trees $T_{1}$ and $T_{2}$ respectively. The previous lemma proves that there are exactly $s(n-s)-1$ independent constraints coming from triples that have indices from both $T_{1},T_{2}$. Putting them together with the aforementioned constraints yield the number of algebraically independent constraints for $T$ as:

[TABLE]

∎

Example 5.7**.**

For the first tree with four terminals in Figure 1 we have:

[TABLE]

while for the second one:

[TABLE]

These group of four equations may be rewritten as:

[TABLE]

and

[TABLE]

respectively. Thus we see that a lesser number of equations suffices.

Remark 5.8**.**

Proposition 5.6 should be understood in a generic sense. To elaborate, it indicates that for an $n$-variate function $F(x_{1},\dots,x_{n})$ one can reduce the total number of ${n\choose 3}$ constraints to ${n-1\choose 2}$; but, in view of the division took place in (5.26), one needs to have $F_{x_{i}}\not\equiv 0$ for every $i$; that is, $F$ should not be independent of any of its variables. This non-vanishing condition is generic: it is open (i.e. persists under small perturbations) and the locus where it fails is of positive co-dimension as it is determined by the union of non-trivial functional equations $F_{x_{i}}\equiv 0$. Such a non-vanishing requirement is necessary because, as a matter of fact, for an arbitrary function $F$ one cannot ignore any of the ${n\choose 3}$ constraints of the form $F_{x_{i}x_{l}}F_{x_{j}}=F_{x_{j}x_{l}}F_{x_{i}}$: Motivated by the non-example (2.3), notice that

[TABLE]

does not satisfy the preceding constraint but satisfies other ones since the mixed second order partial derivative of $F$ w.r.t. any pair of variables other than $(x_{i},x_{j})$, $(x_{i},x_{l})$ and $(x_{j},x_{l})$ is identically zero.

It is also worthy to point out that in the generic situation described above, the number $s(n-s)-1$ in Lemma 5.5 and hence the number ${n-1\choose 2}$ of algebraically independent constraints in Proposition 5.6 cannot be reduced. In other words, there are functions for which all ratios $\frac{F_{x_{i}x_{j}}}{F_{x_{i}}F_{x_{j}}}\,(1\leq i\leq s,s+1\leq j\leq n)$ are the same except $\frac{F_{x_{1}x_{n}}}{F_{x_{1}}F_{x_{n}}}$; e.g.

[TABLE]

whose mixed partial derivatives $F_{x_{i}x_{j}}$ all vanish except $F_{x_{1}x_{n}}$.

5.5. Polynomial function setting

This brief section is devoted to tree representations of polynomials. We are going to see that the local representation that Theorem 2.2 suggests for an $n$-variate polynomial persists throughout $\mathbb{R}^{n}$ and one can always avoid usage of transcendental functions in the representation.

Proposition 5.9**.**

Let $P=P(x_{1},\dots,x_{n})$ be a polynomial satisfying the constraints (2.4) in Theorem 2.2 for a binary tree $T$ with $n$ terminals. Then $P$ has a representation on $T$ that holds on the entirety of $\mathbb{R}^{n}$. Moreover, if $P$ is not independent of any of the variables $x_{1},\dots,x_{n}$, the functions appeared in a representation of $P$ on the tree $T$ must be polynomial as well.

Proof.

We first consider a tree representations of a polynomial $P$ not independent of any of its variables locally around a point, say $\mathbf{0}\in\mathbb{R}^{n}$. We are going to show that the functions appearing in this superposition must be polynomials as well. In the inductive constructions of Proposition 2.1 or that of Section 5.2 (with $P$ in place of $F$) always one of the functions to which the induction hypothesis is applied is obtained by setting some the coordinates to be zero, e.g. $\xi=P(x,y,0)$ or $\xi=P(x_{1},\dots,x_{s};0,\dots,0)$ (check (5.5) or (5.17)) which is a polynomial too. The other functions occurring in the construction, e.g. $g(w,z)$ from the proof of Proposition 2.1 or the function appeared in (5.20), are polynomial as well. The key point is if one locally presents a polynomial as a power series in terms of other polynomials which are non-constant, then the power series must terminate after finitely many terms. This could be easily deduced from the uniqueness of Taylor series. In particular, in (5.19), the single variable functions $u_{k_{s+1},\dots,k_{n}}$ must be a polynomial as both the left hand side and $\xi$ are polynomials of $x_{1},\dots,x_{n}$. Then in (5.20) the function $g(\xi;x_{s+1},\dots,x_{n})$ on a lesser number of variables would be a polynomial as well and so, arguing inductively, its presentation would be entirely in terms of polynomials.

Next, arguing globally, suppose $P=P(x_{1},\dots,x_{n})$ satisfies the constraints (2.4). If $P$ is independent of one the variables $x_{i}$, then the problem reduces to representing polynomials on lesser number of variables by $T$. Thus let us assume $\frac{\partial P}{\partial x_{i}}\not\equiv 0$ for all $1\leq i\leq n$. Then there is a point of $\mathbb{R}^{n}$ at which all partial derivatives are non-zero. By the preceding local discussion, there is a tree representation of $P$ around that point which is necessarily in terms of polynomials. In particular; $P\in\mathcal{F}(T)$, and the representation remains valid on the entirety of $\mathbb{R}^{n}$ since globally defined analytic functions agreeing over a non-vacuous open set must coincide globally. ∎

It is suggestive to abstractly define the space of polynomials subjected to constraints originating from a binary tree. To avoid infinite-dimensional spaces, we put a bound on degrees.

Definition 5.10**.**

The $k^{\rm{th}}$ tree variety ${\rm{Var}}_{T}^{k}$ associated with the binary tree $T$ of $n$ leaves is the Zariski closed subset of the affine variety of $n$-variate (real or complex) polynomials consisting of polynomials $P(x_{1},\dots,x_{n})$ of total degree at most $k$ that satisfy

[TABLE]

for any triple $(x_{i},x_{j},x_{l})$ of leaves of $T$ in which $x_{l}$ is an outsider.

These real or complex varieties could be interesting to study from the algebro-geometric perspective. (Consult [Har77, chap. 1] for the basic notions of algebraic geometry.) They could be thought of as subvarieties of the space ${\rm{Poly}}_{n}^{k}$ of polynomials on $n$ variables $x_{1},\dots,x_{n}$ whose total degree does not exceed $k$. This ambient space is an affine variety of dimension $\binom{k+n}{n}$. Here we find an upper bound on the dimension of the subvariety ${\rm{Var}}_{T}^{k}$.

Proposition 5.11**.**

Let $T$ be a binary tree with $n$ leaves and $k$ a positive integer. The variety ${\rm{Var}}_{T}^{k}$ is of dimension at most $n(k^{2}+1)$.

Proof.

We shall prove this as usual by induction on the number of leaves $n$. For $n=1$ or $n=2$ every polynomial of one or two variables and with total degree at most $k$ could be realized as a tree function. Therefore $\dim{\rm{Var}}_{T}^{k}=k+1$ and $\dim{\rm{Var}}_{T}^{k}=\binom{k+2}{2}$ when $T$ has one or two leaves respectively; and clearly $k+1\leq k^{2}+1$ and $\binom{k+2}{2}\leq 2k^{2}+2$ for all $k\geq 0$.

For the inductive step, following the notation of Theorem 2.2, we denote the left and right sub-trees by $T_{1}$ and $T_{2}$ which respectively have $x_{1},\dots,x_{s}$ and $x_{s+1},\dots,x_{n}$ as leaves where $1\leq s<n$. So the vector

[TABLE]

of coordinates could be written as $\mathbf{x}=\left(\tilde{\mathbf{x}},\tilde{\tilde{\mathbf{x}}}\right)$ where

[TABLE]

Hence a polynomial $P(\mathbf{x})=P(x_{1},\dots,x_{n})\in{\rm{Var}}_{T}^{k}$ may be written as

[TABLE]

where $P_{1}\left(\tilde{\mathbf{x}}\right)=P_{1}(x_{1},\dots,x_{s})$ and $P_{2}\left(\tilde{\tilde{\mathbf{x}}}\right)=P_{2}(x_{s+1},\dots,x_{n})$ admit representations on trees $T_{1}$ and $T_{2}$ respectively, and $g=g(u,v)$ is a polynomial in two variables. As $\deg P\leq k$, the total degree of $P_{1}$ or $P_{2}$ cannot be more than $k$ unless $g$ is independent of $u$ or $v$ in which case one could take $P_{1}$ or $P_{2}$ to be constant as well. So it is safe to assume $P_{1}\in{\rm{Var}}_{T_{1}}^{k}$ and $P_{2}\in{\rm{Var}}_{T_{2}}^{k}$. We condition on the degree of the polynomial $g=g(u,v)$ appeared in (5.29) with respect to its indeterminates as follows:

• •

Suppose $g(u,v)$ is of degree at most one with respect to each indeterminate. Hence $g(u,v)=\alpha u+\beta v+\gamma$ for appropriate scalars $\alpha,\beta,\gamma$ and (5.29) could be rewritten as

[TABLE]

Clearly every polynomial space ${\rm{Var}}_{T^{\prime},k^{\prime}}$ is invariant under affine transformations because one can modify the bivariate polynomial at the root through composition by an affine transformation. Hence the preceding equality exhibits $P(x_{1},\dots,x_{s};x_{s+1},\dots,x_{n})$ as the sum of

[TABLE]

and

[TABLE]

This amounts to an injective morphism $\oplus:{\rm{Var}}_{T_{1}}^{k}\times{\rm{Var}}_{T_{2}}^{k}\rightarrow{\rm{Var}}_{T}^{k}$ defined by addition. The dimension of the range is $\dim{\rm{Var}}_{T_{1}}^{k}+\dim{\rm{Var}}_{T_{2}}^{k}$ which by the induction hypothesis does not exceed

[TABLE]

• •

Suppose either $\deg_{u}g$ or $\deg_{v}g$ is one, say the former. We are going to argue that the situation could again be reduced to the case of affine $g$ and hence the dimension of the corresponding locus is not greater than the dimension calculated above. As $g(u,v)$ is affine with respect to $u$, by the same argument as before one can absorb that affine part and rewrite (5.29) as

[TABLE]

where $P^{\prime}_{1}\in{\rm{Var}}_{T_{1}}^{k}$ is an appropriate affine transform $\alpha.P_{1}+\beta$ of $P_{1}$, $h(v):=g(0,v)$ and $P_{2}$ is a tree polynomial for $T_{2}$. Applying a single variable polynomial to the output of the root of course results in another tree polynomial; i.e. $P^{\prime}_{2}:=h\left(P_{2}(x_{s+1},\dots,x_{n})\right)$ – whose degree is at most $\deg P\leq k$ due to (5.30) – belongs to ${\rm{Var}}_{T_{2}}^{k}$ too. Therefore

[TABLE]

is a sum of elements of ${\rm{Var}}_{T_{1}}^{k}$ and ${\rm{Var}}_{T_{2}}^{k}$. So we simply could revert back to the case that we have already studied.

• •

Finally, suppose $\deg_{u}g$ and $\deg_{v}g$ are both larger than one. Just like the previous part, we could assume that neither $P_{1}$ nor $P_{2}$ is constant. Now in (5.29), the degrees of $P_{1}$ and $P_{2}$ do not exceed $\frac{k}{\deg_{u}g}$ and $\frac{k}{\deg_{v}g}$ respectively as otherwise on the left hand side $\deg P$ would be larger than $k$. Therefore, denoting $\deg P_{1}$ and $\deg P_{2}$ by $j_{1}$ and $j_{2}$, we should have

[TABLE]

Notice that this implies $k\geq 2$. Now the polynomial $g(u,v)$ can only have monomials $u^{c}v^{d}$ with $cj_{1}+dj_{2}\leq k$ as otherwise the degree of $g(P_{1},P_{2})$ would be larger than $k$. Hence $g$ must belong to the following linear space of polynomials

[TABLE]

whose dimension is not larger than the dimension $\binom{k+2}{2}$ of the space of all bivariate polynomials of degree at most $k$. We thus need to consider the ranges of morphisms

[TABLE]

for $j_{1},j_{2}$ such as in (5.31) and the variety ${\rm{V}}_{j_{1},j_{2}}$ as defined in (5.32). We need to argue that the dimension

[TABLE]

of the domain of (5.33) is not greater than $n(k^{2}+1)$. According to the induction hypothesis:

[TABLE]

We conclude that:

[TABLE]

where for the last step we have used the fact that $\frac{3k^{2}}{2}\geq\binom{k+2}{2}$ whenever $k\geq 2$.

∎

Question 5.12**.**

Is there a formula for $\dim{\rm{Var}}_{T}^{k}$ purely in terms of $k$ and the number of leaves of the binary tree $T$?

6. Bit-valued function setting

In this section we investigate the case of bit-valued functions. As outlined in Definition 4.1, the inputs and the output are from $\{0,1\}$; hence we are dealing with functions of the form $\{0,1\}^{n}\rightarrow\{0,1\}$ where $n$ is the number of terminals of the tree $T$ under consideration. There are $2^{2^{n}}$ such functions whereas the number of those which could be implemented on a tree turns out to be far less than this super-exponential number. Theorem 6.1 below and the succeeding corollary provide us with an explicit formula for the number \big{|}\mathcal{F}_{\rm{bin}}(T)\big{|} of such functions which turns out to be only exponential in $n$.

6.1. Discrete tree functions

Before proceeding with enumerating tree functions, it is essential to mention that each elements of $\mathcal{F}_{\rm{bin}}(T)$ is basically a polynomial in $\mathbb{Z}_{2}\left[x_{1},\dots,x_{n}\right]$. This is due to the fact that every function $g:\{0,1\}^{2}\rightarrow\{0,1\}$ assigned to a node can be realized as a bivariate polynomial over the field of two elements $\mathbb{Z}_{2}$:

[TABLE]

We are going to return to this point of view later in this subsection where we formulate binary analogous of the constraint (2.4) using formal differentiation of polynomials in $\mathbb{Z}_{2}\left[x_{1},\dots,x_{n}\right]$.

The following theorem provides us with a recursive construction of binary tree functions.

Theorem 6.1**.**

Let $T_{1}$, $T_{2}$ be binary trees with $n_{1}$ and $n_{2}$ terminals respectively. Connecting them through a node results in a tree $T$ with that node as its root. Labeling the leaves of $T_{1}$ as $\mathbf{x}=\left(x_{1},\dots,x_{n_{1}}\right)$ and the leaves of $T_{2}$ as $\mathbf{y}=\left(y_{1},\dots,y_{n_{2}}\right)$, the space $\mathcal{F}_{\rm{bin}}(T)$ of functions

[TABLE]

*represented by tree $T$ can be described in terms of smaller spaces $\mathcal{F}_{\rm{bin}}(T_{1})$ and $\mathcal{F}_{\rm{bin}}(T_{2})$ as the disjoint union below:

[TABLE]

Proof.

Removing the root of $T$ leaves us with two rooted sub-trees $T_{1},T_{2}$. The function $g:\{0,1\}^{2}\rightarrow\{0,1\}$ at the root then takes in the outputs of the functions $F_{1}=F_{1}(\mathbf{x})$ and $F_{2}=F_{2}(\mathbf{y})$ which are implemented on $T_{1}$ and $T_{2}$ respectively. By (6.1) any such a function can be realized as a bivariate polynomial over $\mathbb{Z}_{2}$. Therefore the right hand side of (6.2) is indeed a subset of $\mathcal{F}_{\rm{bin}}(T)$. We need to show that all functions from $\mathcal{F}_{\rm{bin}}(T)$ have appeared there and moreover, the union is disjoint. We categorize polynomials $g\in\mathbb{Z}_{2}[x,y]$ whose degree w.r.t. each indeterminate is at most one as follows:

1. (i)

$xy,xy+x=x(y+1),xy+y=(x+1)y,xy+x+y+1=(x+1)(y+1);$ 2. (ii)

$xy+1,xy+x+1=x(y+1)+1,xy+y+1=(x+1)y+1,xy+x+y=(x+1)(y+1)-1;$ 3. (iii)

$x+y,x+y+1=(x+1)+y;$ 4. (iv)

$x,x+1;$ 5. (v)

$y,y+1;$ 6. (vi)

$0;$ 7. (vii)

$1.$

These polynomials give rise to all 16 possibilities for $\{0,1\}^{2}\rightarrow\{0,1\}$; cf. (6.1). For any $F_{i}\in\mathcal{F}_{\rm{bin}}(T_{i})(i\in\{1,2\})$, the function $F_{i}+1=F_{i}-1$ obviously belongs to $\mathcal{F}_{\rm{bin}}(T_{i})$ as well. Consequently, for the operation to take place at the root it suffices to consider only one representative from each group as other choices give rise to the same function

[TABLE]

for slightly different inputs $F_{1}$ and $F_{2}$ from $\mathcal{F}_{\rm{bin}}(T_{1})$ and $\mathcal{F}_{\rm{bin}}(T_{2})$. Going with the first polynomial of each group as $g$, we conclude that the expressions below yield all functions in $\mathcal{F}_{\rm{bin}}(T)$ as $F_{1}$ and $F_{2}$ vary in $\mathcal{F}_{\rm{bin}}(T_{1})$ and $\mathcal{F}_{\rm{bin}}(T_{2})$ respectively:

[TABLE]

Of course, in the first three if either $F_{1}$ or $F_{2}$ is constant, the expression would be in the form of one of the last three. Moreover, in the third expression there, changing $F_{1}(\mathbf{x})+F_{2}(\mathbf{y})$ to $\left(F_{1}(\mathbf{x})+1\right)+\left(F_{2}(\mathbf{y})+1\right)$ if necessary, it is safe to assume that the first function is zero at the origin. Therefore, the union on the right of (6.2) is the whole $\mathcal{F}_{\rm{bin}}(T)$. The last step is to show that the subsets in this union are disjoint. Let $(F_{1},F_{2})\in\mathcal{F}_{\rm{bin}}(T_{1})\times\mathcal{F}_{\rm{bin}}(T_{2})$ be a pair of non-constant functions.

1. (a)

The functions $F_{1}(\mathbf{x})F_{2}(\mathbf{y})$, $F_{1}(\mathbf{x})F_{2}(\mathbf{y})+1$ and $F_{1}(\mathbf{x})+F_{2}(\mathbf{y})$ are all non-constant: If otherwise, plugging a $\mathbf{y}_{0}$ with $F_{2}(\mathbf{y}_{0})=1$ implies that $F_{1}(\mathbf{x})$ is constant; a contradiction.

Next, let $(\tilde{F}_{1},\tilde{F}_{2})$ be another pair of non-constant functions picked from $\mathcal{F}_{\rm{bin}}(T_{1})\times\mathcal{F}_{\rm{bin}}(T_{2})$ as well.

1. (b)

The functions $F_{1}(\mathbf{x})F_{2}(\mathbf{y})$, $\tilde{F}_{1}(\mathbf{x})$ and $\tilde{F}_{2}(\mathbf{y})$ are different: Plugging a $\mathbf{y}_{0}$ with $F_{2}(\mathbf{y}_{0})=1$ makes the third one a constant function while the first two are still non-constant. Similarly, plugging an $\mathbf{x}_{0}$ with $F_{1}(\mathbf{x}_{0})=1$ implies $F_{1}(\mathbf{x})F_{2}(\mathbf{y})\not\equiv\tilde{F}_{1}(\mathbf{x})$. 2. (c)

$F_{1}(\mathbf{x})+F_{2}(\mathbf{y})\not\equiv\tilde{F}_{1}(\mathbf{x})$ and $F_{1}(\mathbf{x})+F_{2}(\mathbf{y})\not\equiv\tilde{F}_{2}(\mathbf{y})$ as otherwise, the function $F_{1}(\mathbf{x})$ would be independent of $\mathbf{x}$ and hence constant; a contradiction. 3. (d)

$F_{1}(\mathbf{x})F_{2}(\mathbf{y})\not\equiv\tilde{F}_{1}(\mathbf{x})+\tilde{F}_{2}(\mathbf{y})$ and $F_{1}(\mathbf{x})F_{2}(\mathbf{y})+1\not\equiv\tilde{F}_{1}(\mathbf{x})+\tilde{F}_{2}(\mathbf{y})$: Plugging a $\mathbf{y}_{0}$ with $F_{2}(\mathbf{y}_{0})=0$, the left hand side would be zero or one whereas the right hand side is either $\tilde{F}_{1}(\mathbf{x})$ or $\tilde{F}_{1}(\mathbf{x})+1$, both of them non-constant.

Assuming that furthermore $(F_{1},F_{2})\neq(\tilde{F}_{1},\tilde{F}_{2})$:

1. (e)

We claim that it is impossible for $F_{1}(\mathbf{x})F_{2}(\mathbf{y})-\tilde{F}_{1}(\mathbf{x})\tilde{F}_{2}(\mathbf{y})$ to be a constant function. Assume the contrary. Again, evaluate at a $\mathbf{y}_{0}$ with $F_{2}(\mathbf{y}_{0})=1$. This means either $F_{1}(\mathbf{x})$ or $F_{1}(\mathbf{x})-\tilde{F}_{1}(\mathbf{x})$ should be constant based on whether $\tilde{F}_{2}(\mathbf{y}_{0})$ is zero or one. The former is impossible and the latter implies $F_{1}(\mathbf{x})=\tilde{F}_{1}(\mathbf{x})+\epsilon_{1}$ where $\epsilon_{1}\in\{0,1\}$. Repeating this argument with an $\mathbf{x}_{0}$ where $F_{1}(\mathbf{x}_{0})=1$ requires the other two polynomials to satisfy a similar relation: $F_{2}(\mathbf{y})=\tilde{F}_{2}(\mathbf{x})+\epsilon_{2}$. But then

[TABLE]

must be constant. The case of $\epsilon_{1}=\epsilon_{2}=1$ has been ruled out before in (a); the case of $\epsilon_{1}=\epsilon_{2}=0$ simply means $(F_{1},F_{2})=(\tilde{F}_{1},\tilde{F}_{2})$ which is impossible; and finally, if only one of $\epsilon_{1},\epsilon_{2}$ is one, then either $\tilde{F}_{2}(\mathbf{x})$ or $\tilde{F}_{1}(\mathbf{y})$ must be constant as well. So the aforementioned claim is proven.

So far, we have established the disjointness of the subsets appeared in the union (6.2). Nevertheless, we continue with one more observation that comes in handy soon in Corollary 6.2. Let $(F_{1},F_{2}),(\tilde{F}_{1},\tilde{F}_{2})$ be as in (e) with the additional property of $F_{1}(\mathbf{0})=\tilde{F}_{1}(\mathbf{0})=0$.

1. (f)

$F_{1}(\mathbf{x})+F_{2}(\mathbf{y})\not\equiv\tilde{F}_{1}(\mathbf{x})+\tilde{F}_{2}(\mathbf{y})$ as otherwise $F_{1}(\mathbf{x})-\tilde{F}_{1}(\mathbf{x})$ and $\tilde{F}_{2}(\mathbf{y})-F_{2}(\mathbf{y})$ coincide and hence are both constant functions of the value $F_{1}(\mathbf{0})-\tilde{F}_{1}(\mathbf{0})=0$; that is, $F_{1}=\tilde{F}_{1}$ and $F_{2}=\tilde{F}_{2}$; a contradiction.

∎

Corollary 6.2**.**

For a binary tree with $n$ terminals the number of functions $\{0,1\}^{n}\rightarrow\{0,1\}$ that could be implemented on $T$ is given by:

[TABLE]

Proof.

The formula clearly works for the base cases $n=1,2$ where one gets $4,16$; i.e. the number of functions $\{0,1\}\rightarrow\{0,1\}$ or $\{0,1\}^{2}\rightarrow\{0,1\}$. To do the inductive step, suppose the formula holds for trees $T_{1},T_{2}$ in Theorem 6.1. In the description of $\mathcal{F}_{\rm{bin}}(T)$ as a disjoint union in (6.2), the cardinality of each subset can be easily calculated: For the first three subsets, observations (e) and (f) indicate that different pairs $\left(F_{1}=F_{1}(\mathbf{x}),F_{2}=F_{2}(\mathbf{y})\right)$ result in different functions $F=F(\mathbf{x},\mathbf{y})$. Therefore, both of the first two sets are of cardinality \left(\big{|}\mathcal{F}_{\rm{bin}}(T_{1})\big{|}-2\right).\left(\big{|}\mathcal{F}_{\rm{bin}}(T_{2})\big{|}-2\right) where subtracting $2$ is for excluding constant functions. As for the third set on the right hand side of (6.2), we need to divide by two as well since half of the functions in $\mathcal{F}_{\rm{bin}}(T_{1})$ are zero at $\mathbf{0}$ and the other half are one. Adding up the sizes of the subsets appeared in this partition:

[TABLE]

Substituting \big{|}\mathcal{F}_{\rm{bin}}(T_{1})\big{|} and \big{|}\mathcal{F}_{\rm{bin}}(T_{2})\big{|} from the induction hypothesis:

[TABLE]

that is, (6.3) for the tree $T$ which has $n_{1}+n_{2}$ leaves. ∎

This result once again attests to a recurring theme of this paper: the subset of tree functions is very small compared to whole space of functions. For instance, for $n=4$ we have $2^{16}=65536$ functions but only $520$ of them are expressible in the trees.

Corollary 6.3**.**

The size of the set of functions $\{0,1\}^{n}\rightarrow\{0,1\}$ that are tree functions for a labeled binary rooted tree with $n$ leaves is $O\left(24^{n}\right)$ and is thus $o\left(2^{2^{n}}\right)$.

Proof.

The number of bit-valued functions corresponding to any binary rooted tree with $n$ leaves is $O\left(6^{n}\right)$ by Corollary 6.2. The number of such trees with leaves labeled by $x_{1},\dots,x_{n}$ is clearly $O\left(4^{n}\right)$: It is not hard to see that as a graph it has $2n-1$ vertices and on the other hand, the number of graphs whose vertices are prescribed as a set of size $2n-1$ is $2^{2n-1}$. ∎

The rest of this section is devoted to discrete versions of the constraints we have been working with in previous subsections and their necessity and sufficiency. The ambient space $D_{n}$ of functions $\{0,1\}^{n}\rightarrow\{0,1\}$ in (4.1) can be identified with the following space of polynomials

[TABLE]

that is, polynomials with binary coefficients on $n$ indeterminates whose degree w.r.t. every indeterminate is not larger than one. Notice that (2.4) again furnishes us with a necessary condition for such a polynomial to be represented on a tree $T$ as one can differentiate formally in the ring $\mathbb{Z}_{2}\left[x_{1},\dots,x_{n}\right]$: for any triple of leaves $x_{i},x_{j},x_{l}$ as in Theorem 2.2, the identity

[TABLE]

must hold. Hence (2.3) again serves as a non-example; that is, a binary function of three variables with no tree presentation.

Remark 6.4**.**

A diligent reader with mathematical background may find (6.5) and the derivatives therein problematic since we are moving back and forth between polynomials with coefficients in $\mathbb{Z}_{2}$ and functions with binary inputs; and over a finite field one can easily find pair of polynomials whose values at every single point coincide without being the same in the polynomial sense (which is to have the same coefficients). To address this concern, recall that the space $D_{n}$ of functions $\{0,1\}^{n}\rightarrow\{0,1\}$ has been identified with the space of $n$-variate polynomials in (6.4) via thinking of polynomials as functions $\{0,1\}^{n}\rightarrow\{0,1\}$ by evaluating them at binary vectors of length $n$, and this is an one-to-one correspondence. It is not hard to see that given polynomials $P_{1}(x_{1},\dots,x_{s})$ and $P_{2}(x_{s+1},\dots,x_{n})$ of this form and a function $g:\{0,1\}^{2}\rightarrow\{0,1\}$ – which by (6.1) can always be realized as a polynomial of the same from but on two indeterminates – the $n$-variate polynomial

[TABLE]

is also of the form appeared in (6.4). That is to say, these families of polynomials are closed under tree superpositions. We conclude that the presentation of a binary function $P\in\mathcal{F}_{\rm{bin}}(T)$ as a composition of bivariate functions $\{0,1\}^{2}\rightarrow\{0,1\}$ is indeed a bona fide presentation of an $n$-variate polynomial in terms of bivariate polynomials since once two polynomials from (6.4) give rise to same functions, they coincide as polynomials too. It makes sense to formally take derivatives when we have an equality of polynomials over $\mathbb{Z}_{2}$, hence (6.5) holds for any triple of variables $x_{i},x_{j},x_{l}$ with $x_{l}$ the outsider of the triple.

Remark 6.5**.**

There is furthermore a discrete interpretation of differentiation in this context: Given an $n$-variate polynomial

[TABLE]

from (6.4), $\frac{\partial F}{\partial x_{i}}(\mathbf{x})$ is the same as

[TABLE]

where $\mathbf{e}_{i}$ is the vector from the standard basis whose $i^{\rm{th}}$ entry is $1$ and the rest are [math]; and of course there is no difference between addition and subtraction modulo two. Consequently, we arrive at the following binary version of the constraints (6.5):

[TABLE]

for any triple of variables $(x_{i},x_{j},x_{l})$ in which $x_{l}$ is an outsider.

We conclude the subsection by showing that the constraints on the partial derivatives are sufficient in the binary context too.

Theorem 6.6**.**

Let $T$ be a rooted tree with $n$ leaves and $P(x_{1},\dots,x_{n})$ a polynomial in the form of (6.4); i.e. a polynomial of $n$ variables over $\mathbb{Z}_{2}$ whose degree with respect to each $x_{i}$ is one. Thinking of $P$ as a function $\{0,1\}^{n}\rightarrow\{0,1\}$, it belongs to $\mathcal{F}_{\rm{bin}}(T)$ if and only if for any three leaves $x_{i},x_{j},x_{l}$ of $T$ with $x_{i},x_{j}$ in a rooted sub-tree that does not have $x_{l}$ the identity (6.5) holds.

Proof.

The necessity has been discussed before in this section and we focus on the sufficiency of constraints (6.5). This is going to be achieved by the usual inductive argument. The base case where $T$ has just one or two leaves is clear because every function can be realized as a polynomial (cf. (6.1)) and the condition (6.5) is automatic. For a general $T$, denote the sub-trees to the left and right of the root by $T_{1}$ and $T_{2}$. Suppose their leaves are labeled as

[TABLE]

It is clear that for any two tree functions $P_{i}\in\mathcal{F}_{\rm{bin}}(T_{i})$ ($i\in\{1,2\}$) the functions

[TABLE]

or

[TABLE]

or

[TABLE]

can be represented on the original tree $T$ as they amount to assigning $(a,b)\mapsto a+b$, $(a,b)\mapsto ab$ or $(a,b)\mapsto ab+1$ to the root. Thus it suffices to argue that conversely, every $P\in\mathcal{F}_{\rm{bin}}(T)$ has such a presentation for polynomials $P_{1}=P_{1}(x_{1},\dots,x_{s})$ and $P_{2}=P_{2}(x_{s+1},\dots,x_{n})$ of the same form but on a lesser number of indeterminates that belong to $\mathcal{F}_{\rm{bin}}(T_{1})$ and $\mathcal{F}_{\rm{bin}}(T_{2})$ respectively. Write $P(x_{1},\dots,x_{n})$ as

[TABLE]

Invoking the generalization of (6.5) provided by Lemma 5.1, for any two $1\leq i,j\leq s$ and any non-empty subset $U_{0}\subseteq\{s+1,\dots,n\}$ of the second half of indices, we have

[TABLE]

because $x_{i},x_{j}$ are to the “left” of the node while $x_{u}$’s lie to the “right”. In view of expression (6.10), this implies

[TABLE]

(Keep in mind that $P_{U}$’s are dependent only on the first $s$ indeterminates.) Equating the coefficients of each $\prod_{u\in U}x_{u}$ in both sides results in:

[TABLE]

This simply means that gradient vectors $\left[\frac{\partial P_{U}}{\partial x_{i}}\right]_{1\leq i\leq s}$ of polynomials $P_{U}=P_{U}(x_{1},\dots,x_{s})$ are mutually linearly dependent as $U$ varies among subsets of $\left\{s+1,\dots,n\right\}$. Consequently, any two of them differ solely by their constant terms. Taking the constant term of $P_{U}$’s out of the summation in (6.10) and factoring out, we rewrite the equality as

[TABLE]

where $Q(\overbrace{0,\dots,0}^{s\text{ times}})=0$. If $Q\equiv 0\text{ or }1$, $P$ is dependent only on the last $n-s$ indeterminates and hence is already in the from of (6.7). Otherwise, a high enough partial derivative of it would be $1$. Reversing the roles of indices larger than $s$ and those not greater than $s$, pick a subset $U^{\prime}\subseteq\{1,\dots,s\}$ with

[TABLE]

and indices $s<i^{\prime},j^{\prime}\leq n$. Plugging (6.11) in

[TABLE]

then results in

[TABLE]

Simplifying this, we arrive at

[TABLE]

for any two indices $i^{\prime},j^{\prime}>s$. By the same argument as before, we conclude that up to a constant, polynomials $R(x_{s+1},\dots,x_{n})$ and $\tilde{R}(x_{s+1},\dots,x_{n})$ are scalar multiples of each other. Hence either $R\equiv 0,1$ – in which case $P$ would be the sum of a polynomial of $x_{1},\dots,x_{s}$ and a polynomial of $x_{s+1},\dots,x_{n}$ and hence in the form of (6.7) – or there are $\tilde{\epsilon},\delta\in\{0,1\}$ for which $\tilde{R}=\tilde{\epsilon}R+\delta$. Substituting in (6.11):

[TABLE]

So we have a presentation such as (6.8) when $\delta=0$ and a presentation such as (6.9) when $\delta=1$.

The final step would be to argue that in either of the assignments (6.7), (6.8) or (6.9) $P_{1}$ and $P_{2}$ could be chosen from $\mathcal{F}_{\rm{bin}}(T_{1})$ and $\mathcal{F}_{\rm{bin}}(T_{2})$ respectively. By the induction hypothesis, that is the case if they satisfy the constraint similar to (6.5) but for $T_{1}$ and $T_{2}$. In the latter two presentations, if one of them is identically zero, the other may be chosen to be zero as well and of course the constant function zero can be implemented on any tree. So suppose in (6.8) and (6.9) each of $P_{1}$ and $P_{2}$ is non-zero for certain inputs $\tilde{\mathbf{x}}_{0}\in\{0,1\}^{s}$ or $\tilde{\tilde{\mathbf{x}}}_{0}\in\{0,1\}^{n-s}$. Pick $\tilde{\mathbf{x}}_{0},\tilde{\tilde{\mathbf{x}}}_{0}$ arbitrarily in the case of (6.7). Now evaluating at such points gives expressions of $P_{1}$ and $P_{2}$ as

[TABLE]

for some appropriate $\alpha,\beta\in\{0,1\}$. Thus $P_{1}$ and $P_{2}$ must satisfy (6.5) once the leaves $x_{i},x_{j},x_{l}$ there are all from the left sub-tree $T_{1}$ or from the right sub-tree $T_{2}$. ∎

6.2. A metric on the set of labeled binary trees

The spaces of discrete functions associated with binary trees in §6.1 could be employed to quantify how different two such trees are. It must be mentioned that the difference is not merely caused by combinatorially distinct underlying graphs, but different labeling schemes may also give rise to different function spaces; keep in mind that in our context binary trees always come with leaves labeled by coordinate functions. Hence we are dealing with the following set:

Definition 6.7**.**

By ${\rm{Tree}}_{n}$ we denote the set of all binary rooted trees with $n$ terminals labeled by $x_{1},\dots,x_{n}$ modulo isomorphism of rooted trees.

Modding out by isomorphisms is because operations such as swapping labels of two sibling leaves leave the function space unchanged. By abuse of notation, we denote both a binary rooted tree and its class in ${\rm{Tree}}_{n}$ by the same symbol $T$. Figure 8 illustrates elements of ${\rm{Tree}}_{4}$ where labels come from the set $\{x,y,z,w\}$ rather than $\{x_{1},x_{2},x_{3},x_{4}\}$.

Symmetric difference of discrete functions spaces furnishes ${\rm{Tree}}_{n}$ with a natural metric:

Definition 6.8**.**

The distance between two trees, $T_{1}$ and $T_{2}$, is defined by the symmetric difference metric:

[TABLE]

Keep in mind that the sizes of $\mathcal{F}(T_{1})$ and $\mathcal{F}(T_{2})$ are the same and dependent only on $n$ by the virtue of Corollary 6.2.

Each constraint imposed in (2.4) or (6.5) on elements of $\mathcal{F}(T)$ or $\mathcal{F}_{\rm{bin}}(T)$ reveals something about the combinatorics of the tree $T$: the most immediate common ancestor of leaves $x_{i},x_{j}$ is not an ancestor of the leaf $x_{l}$. The proposition below shows that one can indeed reconstruct $T$ from the knowledge of function spaces $\mathcal{F}_{\rm{bin}}(T)$ or $\mathcal{F}(T)$. This result is interesting in its own sake as Corollary 6.2 says that the cardinality of $\mathcal{F}_{\rm{bin}}(T)$ is only dependent on the number of leaves of $T$; or in the analytic setting, in the sense of Proposition 5.6 the number of algebraically independent constraints has nothing to do with the morphology of the tree $T$ with $n$ leaves.

Proposition 6.9**.**

A rooted binary $T$ with $n$ terminals could be recovered from the corresponding set $\mathcal{F}_{\rm{bin}}(T)$ of bit-valued functions $\{0,1\}^{n}\rightarrow\{0,1\}$. The same is true for the function space $\mathcal{F}(T)$.

Proof.

As usual, we label the terminals of $T$ by $x_{1},\dots,x_{n}$ so that elements of $\mathcal{F}_{\rm{bin}}(T)$ can be thought of as polynomials in $\mathbb{Z}_{2}\left[x_{1},\dots,x_{n}\right]$ whose degrees w.r.t. every indeterminate is at most one. Given any three different leaves $x_{i_{s}}\,(s\in\{1,2,3\})$, there is one of them which is farther apart from the other two in the sense that it lies outside a rooted sub-tree which contains the other two. The knowledge of the outsider leaf in every possible triple of leaves completely determines the combinatorics of the tree. The constraints (6.5) indicate that

[TABLE]

if $x_{i_{3}}$ is the one which is separated from $x_{i_{1}},x_{i_{2}}$. Consequently, it suffices to come up with a polynomial that satisfies the constraint above but not the analogous ones

[TABLE]

corresponding to situations where $x_{i_{1}}$ or $x_{i_{2}}$ is the outsider of $\{x_{i_{1}},x_{i_{2}},x_{i_{3}}\}$. The polynomial $\left(x_{i_{1}}+x_{i_{2}}\right)x_{i_{3}}$ has all of these properties. Moreover, it obviously belongs to $\mathcal{F}_{\rm{bin}}(T)$ if $x_{i_{3}}$ is separated from $x_{i_{1}},x_{i_{2}}$ via a rooted sub-tree: the node which is the root of this sub-tree gets $x_{i_{1}}$ and $x_{i_{2}}$ as inputs and adds them. The resulting sum is then passed to the root of $T$ along with $x_{i_{3}}$, and then a multiplication takes places at the root.

The proof above works equally well in the analytic setting as one can think of $\left(x_{i_{1}}+x_{i_{2}}\right)x_{i_{3}}$ as an analytic function $\mathbb{R}^{n}\rightarrow\mathbb{R}$ rather than a binary one $\{0,1\}^{n}\rightarrow\{0,1\}$. ∎

Corollary 6.10**.**

The function ${\rm{d}}:{\rm{Tree}}_{n}\times{\rm{Tree}}_{n}\rightarrow[0,1]$ from (6.12) defines a metric on the set ${\rm{Tree}}_{n}$ of labeled binary rooted trees.

Proof.

Symmetry and triangle inequality trivially hold. Proposition 6.9 implies that ${\rm{d}}$ is a metric rather than a pseudo-metric as ${\rm{d}}(T_{1},T_{2})=0$ implies $T_{1}=T_{2}$. ∎

Example 6.11**.**

This example, like Example 5.7, investigates binary trees with four terminals. Here, we calculate the distances between some of the binary trees illustrated in Figure 8. By the virtue of (6.3), the total size of each function space is $520$ when $n=4$. Let us first consider two different trees e.g. those of Figure 1 which appear as $T_{3},T_{4}$ in Figure 8. We need to compute how many four variable functions $F(x,y,z,w)$ admitting a representation by the symmetric tree $T_{3}$ can also be represented by the asymmetric one $T_{4}$. Based on the results of §6.1 and Theorem 6.6, $F(x,y,z,w)$ could be thought of as a polynomial whose degree with respect to each indeterminate is one (i.e. is in the form (6.4)) that moreover, satisfies (5.28). It has a representation by the latter tree if and only if it satisfies the constraints (5.27). Comparing these two groups of equations, one only requires the identities below to hold in the ring $\mathbb{Z}_{2}[x,y,z,w]$:

[TABLE]

The description (6.2) of tree functions in terms of sub-trees comes in handy now. Constant functions and bivariate functions of $x,y$ or of $z,w$ could definitely be represented by the asymmetric tree. Next, we determine which functions of the form $F_{1}(x,y)F_{2}(z,w)$, $F_{1}(x,y)F_{2}(z,w)+1$ or $F_{1}(x,y)+F_{2}(z,w)$ with $F_{1},F_{2}$ being non-constant polynomials of the form (6.4) satisfy (6.13). Plugging the first two in (6.13) yields

[TABLE]

Either $\frac{\partial F_{1}}{\partial x}$ or $\frac{\partial F_{1}}{\partial y}$ is non-zero. Thus (6.13) boils down to $\frac{\partial^{2}F_{2}}{\partial z\partial w}F_{2}=\frac{\partial F_{2}}{\partial z}\frac{\partial F_{2}}{\partial w}$; and this holds if and only if $F_{2}(z,w)$ is a product of linear polynomials of $z,w$; that is, $F_{2}(z,w)$ is one of the followings:

[TABLE]

Next, writing (6.13) for $F=F_{1}(x,y)+F_{2}(z,w)$ (where $F_{1}(0,0)=0$ as in (6.2)) yields

[TABLE]

Since $F_{1}(x,y)$ is non-constant, this means $\frac{\partial^{2}F_{2}}{\partial z\partial w}=0$, i.e. $F_{2}$ is one of the affine polynomials below:

[TABLE]

Now for each subset appearing in the disjoint union (6.2) we know how many functions admit representations by both symmetric and asymmetric trees in Figure 1. This enables us to calculate the size of the intersection of the corresponding function spaces as

[TABLE]

Hence the desired distance is $\frac{1}{520}(520-296)\approx 0.43$.

We conclude the example by finding the distance for a case where a tree is considered with two different labels. Take the symmetric trees $T_{2}$ and $T_{3}$ in the top row of Figure 8. Invoking the description (6.2) of tree functions $F(x,y,z,w)$ in terms of sub-trees again, it is not hard to see that the only tree functions in common are those in the product form

[TABLE]

or in the sum form

[TABLE]

This is due to the fact that a polynomial identity such as $F_{1}(x,y)F_{2}(z,w)\equiv\tilde{F}_{1}(x,z)\tilde{F}_{2}(y,w)$ (resp. $F_{1}(x,y)+F_{2}(z,w)\equiv\tilde{F}_{1}(x,z)+\tilde{F}_{2}(y,w)$) requires all constituent parts (assumed to be non-constant) to be product (resp. sum) of single variable polynomials. Hence the cardinality of the intersection is

[TABLE]

where the summands respectively account for the following possibilities for $F$:

• •

all four polynomials in (6.14) are of degree one;

• •

three of polynomials in (6.14) are of degree one and the other is constant $1$;

• •

two of polynomials in (6.14) are of degree one and the other two are constant $1$;

• •

$F$ is a linear combination of $1,x,y,z,w$ over $\mathbb{Z}_{2}$ like (6.15).

Consequently, we arrive at

[TABLE]

7. Application to neural networks

The goal of this section is to adapt the techniques used so far to study feed-forward neural networks. In the first subsection, we present a method for passing from such a neural network to a rooted tree. In the later subsections we try to imitate arguments of §5 and §6 for more general trees. Two main difficulties come up then:

• •

the number of leaves is probably larger than the number of variables of the function to be implemented; in other words, different leaves could correspond to the same variable;

• •

trees are not binary anymore.

In this section trees are, as always, rooted with their leaves labeled. One can adapt the terminology of §4 to non-binary trees with no difficulty. The number of leaves/terminals is again denoted by $n$. Other vertices, called nodes/branch points/non-terminals, have at least two and at most $c$ successors where $c$ is the maximum number of children/successors of a node.888For the sake of inductive argument, we consider a single vertex to be a tree with only one leaf.

7.1. From neural networks to trees

Trees are just one particular architecture for neural networks. Nevertheless, they could serve as building blocks. A neural network is made up of multiple layers stacking atop of each other with each layer containing several nodes. Denoting the layers by $L_{1}$ up to $L_{n}$, the nodes in the first layer $L_{1}$ correspond to the inputs. Associated with each node of the $i^{\rm{th}}$ layer is a function which as its inputs takes the outputs of a subset of nodes (at least two) in the $i-1^{\rm{th}}$ layer. The first layer takes the input and after that the functions associated to the nodes in the next layer compute the values for this layer. This continues up to the last layer where the output of the neural network is returned at the root. Therefore, once again superpositions of functions take place. Unless otherwise stated, we assume functions applied at nodes to be either analytic or bit-valued.

Tree functions are examples of functions computed via neural networks. Conversely, a feed-forward neural network $N$ may be converted to its corresponding TENN (the Tree Expansion of the Neural Network) which is a (not necessarily binary) rooted tree $T$. This construction is best demonstrated in Figure 4. Abstractly speaking, each vertex of the tree $T$ represents a path starting from a vertex of the neural network $N$ and ascending through the layers until it reaches the root of $N$. Given such a path, for any two sub-paths to the root determined by successive vertices of $N$ the corresponding vertices of $T$ must be connected.

We should keep in mind that in this expansion nodes and leaves might be repeated (both for the input layer and upper layers). Consequently, unlike the convention in §4, we are dealing with trees for which different leaves may share a label.

7.2. Trees with repeated labels; a toy example

In Theorem 2.2 we characterized analytic tree functions via a system of PDEs. The working assumption in that theorem and also throughout §5, §6 was that the leaves of the tree correspond to different variables. On the contrary, as discussed in the preceding section, the tree resulting from expanding a neural network is not necessarily binary and there are probably different leaves labeled by the same variable.

To demonstrate how cumbersome a PDE constraint may get in presence of repetition, let us go back to the toy example of a function of $x,y,z$ from §5.1, but this time in the form of

[TABLE]

rather than (5.1). This is a superposition of function $f(x,y),h(x,z)$ and $g(u,v)$ and is illustrated in Figure 10. Switching to the subscript notation for partial derivatives, we try the imitate the usage of chain rule in §5.1 which resulted in constraint (2.2). Differentiating yields:

[TABLE]

and therefore, the linear combination

[TABLE]

Notice that weights $A,B$ in the linear combination above do not depend on $z$ and $y$. Taking higher partial derivatives of (7.2) with respect to $y,z$ results in a description of derivatives of the form $F_{xy\dots yz\dots z}$ as linear combinations of functions such as $F_{y\dots y}$ and $F_{z\dots z}$. But assuming that we have differentiated up to $n$ times, there are ${n+2\choose 2}$ partial derivatives such as $F_{xy\dots yz\dots z}$ in which the total order of differentiation with respect to $y,z$ is at most $n$. On the other side of the linear combination, the weights could only be partial derivatives $A_{y\dots y}$ and $B_{z\dots z}$ of order not greater than $n$. But there are $2(n+1)$ such terms. This implies that there is a linear dependency of the aforementioned partial derivatives of $F$ provided that $n$ is large enough. And a linear dependency could always be described as the vanishing of determinants.

For example, by differentiating (7.2) up to three times, we arrive at:

[TABLE]

Proof of Proposition 2.3.

The vector on the left of (7.3) is in the column space of the $7\times 6$ matrix appeared on the right. Joining them results in a $7\times 7$ matrix whose columns are linearly dependent and its determinant thus must be zero. ∎

Article [Arn09a] alludes to the function $xy+yz+zx$ as a function that is not a superposition of the form (7.1) of continuous bivariate functions. Notice that this function satisfies the condition of Proposition 2.3; hence that condition is not sufficient. Example below addresses a similar issue with our recurring non-example.

Example 7.1**.**

The familiar function $xyz+x+y+z$ from (2.3) failed to satisfy the constraint (2.2) while it is not hard to see that it is a solution to the PDE from Proposition 2.3. Here we are going to show that it cannot be represented globally as a superposition of the form (7.1) (and in retrospect as a superposition of the form (5.1)) even in the continuous category. Aiming for a contradiction, suppose

[TABLE]

for globally defined continuous functions $f,g,h:\mathbb{R}^{2}\rightarrow\mathbb{R}$. Substituting $y$ with $\frac{-1}{x}$ yields

[TABLE]

for any $x,y,z$ with $x\neq 0$. Let $x_{0}\neq 0$ and $z_{0}$ be arbitrary. The function $z\mapsto h(x_{0},z)$ could not be constant over a non-degenerate interval because then

[TABLE]

would be independent of $z$ on some non-degenerate rectangular region of the plane. Hence there is an $\epsilon>0$ such that the image of $z\mapsto h(x_{0},z)$ contains an open interval of the form $(a-2\epsilon,a+2\epsilon)$ where $a:=h(x_{0},z_{0})$. By continuity, for sufficiently small $\delta>0$ the image of $z\mapsto h(x_{1},z)$ contains the interval $J:=(a-\epsilon,a+\epsilon)$ provided that $|x_{0}-x_{1}|<\delta$. Now for any such an $x_{1}$ the function $g$ is constant on the vertical segment

[TABLE]

because of (7.5). Hence if

[TABLE]

is not constant in vicinity of $x_{0}$, then $g(u,v)$ would be dependent only on its first variable for $(u,v)$ from a non-degenerate rectangle containing $\left(f\left(x_{0},\frac{-1}{x_{0}}\right),a\right)=\left(f\left(x_{0},\frac{-1}{x_{0}}\right),h(x_{0},z_{0})\right)$. But for $(x,y,z)$ close enough to $\left(x_{0},\frac{-1}{x_{0}},z_{0}\right)$ the point

[TABLE]

and hence $g(f(x,y),h(x,z))$ is dependent only on $x,y$ as $(x,y,z)$ varies in a sufficiently small neighborhood of $\left(x_{0},\frac{-1}{x_{0}},z_{0}\right)$. This is of course impossible since the right hand side of (7.4) is not independent of $z$ when $y\neq\frac{-1}{x}$. Therefore, (7.6) must be constant over an interval $(x_{0}-\delta^{\prime},x_{0}+\delta^{\prime})$ where $0<\delta^{\prime}\leq\delta$. We next arrive at a contradiction: Pick an $x_{1}\neq x_{0}$ form this interval. By the previous discussion, there is a number $z_{1}$ such that

[TABLE]

But then

[TABLE]

which by (7.5) requires $x_{0}-\frac{1}{x_{0}}$ and $x_{1}-\frac{1}{x_{1}}$ to be the same; a contradiction as $x\mapsto x-\frac{1}{x}$ is injective.

The preceding discussion indicates that in the case of trees with repeated labels (hence, following the discussion in §7.1, the case of neural networks) numerous PDE constraints originated from the linear algebra could be imposed on tree functions. In Theorem 2.2 we observed that in the case of distinct labels, a finite number of PDEs are enough to pin down analytic tree functions. This promotes us to ask for a generalization:

Question 7.2**.**

Is it true that analytic superpositions corresponding to any given architecture of neural networks may be characterized as solutions to a finite system of PDEs?

7.3. Trees with repeated labels; estimates

After realizing the functions produced by neural networks as tree functions for trees with repeated labels, we next proceed to invoke our results to study tree function spaces in presence of repeated labels. In the context of bit-valued functions, obtaining a precise count of discrete functions such as (6.3) is implausible. Nevertheless, one could easily bound the number of such functions. Working with the notation fixed in the beginning of this section and fixing a tree $T$ with $n$ leaves, the tree functions under consideration are of the form

[TABLE]

where $x_{1},\dots,x_{p}$ are the labels of the leaves of $T$; so $p\leq n$. These functions are superpositions of functions of the form $\{0,1\}^{i}\rightarrow\{0,1\}$ each assigned to a node where $2\leq i\leq c$ is the number of inputs that the node receives. Hence a very crude overestimate for $\left|\mathcal{F}_{\rm{bin}}(T)\right|$ is given by

[TABLE]

The number of nodes/non-terminal vertices of $T$ could be easily bounded in terms of $n$:

Lemma 7.3**.**

A rooted tree $T$ with $n$ leaves has at most $n-1$ nodes with equality if and only if $T$ is binary.

Proof.

From the basic graph theory ([HHM08, chap. 1]) the number of edges of the underlying graph of $T$ is simultaneously the number of vertices minus one and half the sum of degrees of all vertices. The former is

[TABLE]

while the latter is at least

[TABLE]

due to the fact that the degree of every node other than the root itself is at least three (it has one predecessor and at least two successors). Therefore:

[TABLE]

The equality occurs exactly when every non-root node has only two successors; i.e. the rooted tree $T$ is binary. ∎

This lemma yields $\left(2^{2^{c}}\right)^{n-1}$ as an upper bound for the size of the discrete TFS $\mathcal{F}_{\rm{bin}}(T)$ of the tree $T$ with $n$ leaves appeared before. Invoking the ideas developed in the proof of Theorem 6.1, this bound could be improved: Assuming that the root of $T$ is of degree $m$ ($2\leq m\leq c$), as its inputs it receives the outputs of $m$ tree functions $F_{1},\dots,F_{m}$ implemented on the smaller sub-trees emanating from the root. These outputs are then passed to a function $g:\{0,1\}^{m}\rightarrow\{0,1\}$ at the root and at last, the procedure results in $f:=g\left(F_{1},\dots,F_{m}\right)$ as our tree function. The key point is that it is not necessary to consider all $2^{m}$ possibilities for $g$ since if, for instance, one modifies $g(x_{1},x_{2},\dots,x_{m})$ as $g(x_{1}+1,x_{2},\dots,x_{m})$, then feeding the root with the bit-valued function $F_{1}-1=F_{1}+1$ instead of $F_{1}$ results in the same function $f$; notice that $F_{1}+1$ is obviously a tree function for the same sub-tree. In other words, identifying the set of functions $\left\{0,1\right\}^{m}\rightarrow\left\{0,1\right\}$ with the set

[TABLE]

of binary polynomials (just like what we did in (6.4)), we are going to argue that the set above partitions to certain equivalence classes such that the choices for the function assigned to the root could be narrowed down to a set of representatives of these classes; hence a fewer number of choices. This is a direct generalization of what we did for binary trees in the proof of Theorem 6.1 where we divided functions $\{0,1\}^{2}\rightarrow\{0,1\}$ into seven groups. More precisely, two binary polynomials $g(x_{1},\dots,x_{m})$ and $\tilde{g}(x_{1},\dots,x_{m})$ of the form above (meaning of degree at most one with respect to each indeterminate) are regarded to be equivalent if one is obtained from the other by changing $x_{i}$ to $x_{i}+1$ for a subset of indeterminates. We shall need the number of such equivalence classes. This is the content of the lemma below.

Lemma 7.4**.**

The number of classes of the equivalence relation on the set

[TABLE]

defined above is $2^{2^{m-1}-m}\left(2^{2^{m-1}}+2^{m}-1\right)$.

Proof.

The proof is an application of Burnside’s lemma [HHM08, §2.7]. These equivalence classes are orbits of an action of the group $(\mathbb{Z}_{2})^{m}$ on the subset (7.8) of $\mathbb{Z}_{2}\left[x_{1},\dots,x_{m}\right]$ defined as

[TABLE]

for any $m$-tuple $\boldsymbol{\delta}:=\left(\delta_{1},\dots,\delta_{m}\right)$ of [math]’s and $1$’s. Burnside’s lemma gives the number of orbits as the sum of the number of polynomials in (7.8) fixed by $\boldsymbol{\delta}$ as $\boldsymbol{\delta}$ varies in $(\mathbb{Z}_{2})^{m}$ divided by the cardinality $2^{m}$ of $(\mathbb{Z}_{2})^{m}$. It is not hard to see that an non-identity element $\boldsymbol{\delta}$ fixes exactly half of the polynomials; e.g. $\boldsymbol{\delta}=(1,0,\dots,0)\in(\mathbb{Z}_{2})^{m}$ fixes only those members of (7.8) in which $x_{1}$ does not appear and there are $2^{2^{m-1}}$ of them. The general situation for a group element

[TABLE]

in which one $\delta_{i}$, say $\delta_{1}$, is non-zero could be reduced to the aforementioned case by the linear change of coordinate

[TABLE]

due to the fact that $\boldsymbol{\delta}$ takes $\tilde{x}_{1}$ to $\tilde{x}_{1}+1$ while $\tilde{x}_{2},\dots,\tilde{x}_{m}$ are preserved. We conclude that the number of orbits is:

[TABLE]

∎

Now we arrive at the following improvement of the bound $\left(2^{2^{c}}\right)^{n-1}$ derived before:

Proposition 7.5**.**

Let $T$ be a tree with $n$ leaves whose nodes have at most $c\geq 2$ children. Then

[TABLE]

Proof.

In the base case, when $n=1$, there are exactly four functions $\{0,1\}\rightarrow\{0,1\}$ and the inequality is clearly satisfied. For the inductive step and with notation as before, removing the root of $T$ leaves us with $m$ sub-trees $T_{1},\dots,T_{m}$ where $2\leq m\leq c$. Denoting numbers of their leaves by $n_{1},\dots,n_{m}$, one has $n_{1}+\dots+n_{m}=n$ and by the induction hypothesis

[TABLE]

for all $1\leq i\leq m$. Lemma 7.4 now implies that

[TABLE]

∎

Corollary 7.6**.**

A tree $T$ with $n$ leaves whose nodes have no more than $c$ children satisfies $\left|\mathcal{F}_{\rm{bin}}(T)\right|=O\left(\gamma_{c}^{n}\right)$ where

[TABLE]

is a number smaller than $2^{2^{c}}$.

Proof.

Proposition 7.5 indicates that the number of bit-valued tree functions implemented on $T$ is

[TABLE]

When $c\geq 3$ the base $2^{2^{c-1}-c+2}\left(2^{2^{c-1}}+2^{c}-1\right)$ is less than the number $2^{2^{c}}$ appeared in the rudimentary bound (7.7). For $c=2$ the tree is binary and we could use the bound $O\left(6^{n}\right)$ on $\left|\mathcal{F}_{\rm{bin}}(T)\right|$ that Corollary 6.2 provides. ∎

We finish with two applications to trees with repeated labels and neural networks. First, suppose the goal is to implement all functions $\{0,1\}^{p}\rightarrow\{0,1\}$ on a rooted tree whose leaves are labeled by $x_{1},\dots,x_{p}$ and let $c$ be the maximum possible number of the children of a node. The essence of the corollary below is that for this goal to be achieved, fixing $c$ the number $n$ of the terminals must be $O(2^{p})$ as $p$ grows.

Corollary 7.7**.**

Let $T$ be a tree with $n$ leaves labeled by variables $x_{1},\dots,x_{p}$ whose nodes have at most $c\geq 2$ children and let $\gamma_{c}$ be as in (7.9). Every function $F:\{0,1\}^{p}\rightarrow\{0,1\}$ could be implemented on this tree only if

[TABLE]

for $c=2$, and

[TABLE]

if $c\geq 3$.999All logarithms are in base $2$.

Proof.

The number $2^{2^{p}}$ of functions $\{0,1\}^{p}\rightarrow\{0,1\}$ cannot be greater than $\left|\mathcal{F}_{\rm{bin}}(T)\right|$. When $c=2$, we use formula (6.3) for this cardinality while for $c\geq 3$, we invoke the upper bound from Proposition 7.5. ∎

We finally apply the previous results to tree expansions of neural networks in order to compare different architectures.

Definition 7.8**.**

For a neural network $N$, the number of leaves in the expanded-tree is denoted by $L(N)$. The corresponding set of bit-valued functions implemented on $N$ is shown by $\mathcal{F}_{\rm{bin}}(N)$.

Theorem 7.9**.**

For a neural network $N$ in which each node is connected to at most $c$ nodes from the previous layer one has

[TABLE]

Proof.

Apply Corollary 7.6 to the TENN that corresponds to $N$. ∎

Acknowledgements. The authors are grateful to Mehrdad Shahshahani for proposing Remark 5.3, to David Rolnick for reading an early draft of this paper, to Mohammad Ahmadpoor, Ari Benjamin, Ben Lansdell, Pat Lawlor, Samantha Ing-Esteves and Nidhi Seethapathi for helpful conversations, to the organizers of The Science of Deep Learning colloquium where a poster based on this work was presented, to the users contributed to answering a question we posted on the MathOverflow network about representations of ternary functions in terms of bivariate functions and to the NIH for funding (R01MH103910).