Nonparametric Functional Approximation with Delaunay Triangulation

TL;DR

The paper introduces a differentiable nonparametric method called Delaunay triangulation learner (DTL) that partitions feature space into simplices for functional approximation, combining geometric optimality with linear modeling.

Contribution

It presents a novel DTL algorithm that leverages Delaunay triangulation for nonparametric function approximation, with theoretical analysis and empirical comparison.

Findings

DTL effectively partitions feature space into simplices.

Theoretical properties of DTL are rigorously analyzed.

DTL outperforms some existing learners in numerical studies.

Abstract

We propose a differentiable nonparametric algorithm, the Delaunay triangulation learner (DTL), to solve the functional approximation problem on the basis of a $p$-dimensional feature space. By conducting the Delaunay triangulation algorithm on the data points, the DTL partitions the feature space into a series of $p$-dimensional simplices in a geometrically optimal way, and fits a linear model within each simplex. We study its theoretical properties by exploring the geometric properties of the Delaunay triangulation, and compare its performance with other statistical learners in numerical studies.

Equations

$$\mathcal{S}(x,\mathcal{X})=\underset{\mathcal{S}\in \mathcal{D}(\mathcal{X})}{\mathrm{argsup}}I\left(x\in \mathcal{S}\right),$$

$$F_D(x;\Psi )=\{\begin{array}{ll}g(x;\mathcal{S}(x,\mathcal{X}),\Psi ),& \text{if x∈H(X)},\\ y(x(\mathcal{X})),& \text{if x∈H(X)},\end{array}$$

$$g(x_i;\mathcal{S}(x_i,\mathcal{X}),\Psi )={\psi }_i,i=1,\dots ,n.$$

$${\Psi }^{\ast }=\underset{\Psi }{\mathrm{arginf}}L(y,F_D(x;\Psi ))+\lambda R(F_D(x;\Psi )),$$

$$\{\begin{array}{ll}& ⟨n(\mathcal{S}),({\mathbb{G}}^{\top },{\mathbb{G}}^{\top }\mathbb{G}{)}^{\top }⟩=0,\\ & ⟨n(\mathcal{S})[1:p],{\mathbb{G}}^{\top }⟩=0,\\ & \Vert n(\mathcal{S})\Vert =1,\\ & ⟨n(\mathcal{S}),{\mathbf{e}}_{p+1}⟩\ge 0,\end{array}$$

$${\Delta }_{x_i}$$

$${\Delta }_{{\psi }_i}$$

$$n(\mathcal{S})=\frac{(-{\mathbb{G}}^{\top },1{)}^{\top }}{\Vert (-{\mathbb{G}}^{\top },1{)}^{\top }\Vert }.$$

$$K(\mathbf{v})={\left(\genfrac{}{}{0ex}{}{|\mathcal{N}(\mathbf{v})|}{2}\right)}^{-1}\sum _{\begin{array}{c}k<j\\ {\mathcal{S}}_k,{\mathcal{S}}_j\in \mathcal{N}(\mathbf{v})\end{array}}\measuredangle \left(n({\mathcal{S}}_k),n({\mathcal{S}}_j)\right),$$

$$R\left(F_D(x;\Psi )\right)$$

$$H(\mathbb{X},\mathcal{X},\mathcal{T})=\sum _{i=1}^{p+1}{\lambda }_i\Vert {\mathbb{X}}_{(i)}-\mathbb{X}{\Vert }^2,\text{ s.t. ∑i=1p+1λi=1, ∑i=1p+1λiX(i)=X,}$$

$$\mathbb{E}[H(\mathbb{X},\mathcal{X},\mathcal{D})]=\underset{\mathcal{T}}{\inf }\mathbb{E}[H(\mathbb{X},\mathcal{X},\mathcal{T})],$$

$$\tilde{H}(\mathbb{X},{\mathcal{X}}_{p+1})=\sum _{i=1}^{p+1}{\lambda }_i\Vert {\mathbb{X}}_i-\mathbb{X}{\Vert }^2,\text{ s.t. ∑i=1p+1λi=1, ∑i=1p+1λiXi=X.}$$

$$H(\mathbb{X},\mathcal{X},\mathcal{D})=\underset{{\mathcal{X}}_{p+1}\subset \mathcal{X}}{\inf }\tilde{H}(\mathbb{X},{\mathcal{X}}_{p+1})\le H(\mathbb{X},\mathcal{X},\mathcal{T})$$

$$\mathbb{E}[H(\mathbb{X},\mathcal{X},\mathcal{D})]=\underset{\mathcal{T}}{\inf }\mathbb{E}[H(\mathbb{X},\mathcal{X},\mathcal{T})].$$

$$\mathbb{E}\Vert {\mathbb{X}}_{(1)}-\mathbb{X}{\Vert }^2\le c_p{\left(\frac{1}{n+1}\right)}^{2/p},$$

$$\mathbb{E}\Vert {\mathbb{X}}_{(1)}-\mathbb{X}{\Vert }^2\le \frac{2}{n+1}.$$

$$\mu \left({\cup }_{i=1}^{n+1}{B}_i\right)\le (1+\sqrt{p}{)}^p,$$

$$\sum _{i=1}^{n+1}{V}_p{\left(\frac{r_i}{2}\right)}^p\le (1+\sqrt{p}{)}^p.$$

$${\left(\frac{1}{n+1}\sum _{i=1}^{n+1}{r}_i^2\right)}^{p/2}$$

$$\frac{1}{n+1}\sum _{i=1}^{n+1}{r}_i^2\le \frac{2}{n+1}.$$

$$\mathbb{P}(E_n^c)\le {\left(1-\underset{j\in \{1,\dots ,2^p\}}{\min }{\int }_{U_j}f(x)dx\right)}^n\to 0,$$

$$\mathbb{P}(H_n)$$

$$n^{1/p}\mathbb{E}T(\mathbb{X})\to \beta {\int }_{[0,1{]}^p}f(x{)}^{(p-1)/p}dx,$$

$$T(\mathbb{X})=I(H_n)\tilde{T}(\mathbb{X})+(1-I(H_n))\Vert \mathbb{X}-{\mathbb{X}}_{(1)}\Vert ,$$

$$\mathbb{E}\Vert \mathbb{X}-{\mathbb{X}}_{(1)}{\Vert }^2$$

$$\frac{\mu (B(x,\rho ))}{V_p{\rho }^p}\to f(x),\mathrm{as}\rho \downarrow 0,$$

$${\left\{1-\mu \left(B\left(x,\frac{\sqrt{t}}{n^{1/p}}\right)\right)\right\}}^n\to \exp (-f(x)V_p{t}^{p/2}),\mathrm{as}n\to \infty .$$

$$\underset{n\to \infty }{\lim }\inf n^{2/p}{\int }_0^{\infty }\mathbb{P}\left\{\Vert \mathbb{X}-{\mathbb{X}}_{(1)}{\Vert }^2>t\right\}dt$$

$$n^{2/p}{\int }_0^{\infty }\mathbb{P}\left\{\Vert \mathbb{X}-{\mathbb{X}}_{(1)}{\Vert }^2>t\right\}dt$$

$$\underset{n\to \infty }{\lim }\sup n^{2/p}\mathbb{E}\left\{\Vert \mathbb{X}-{\mathbb{X}}_{(1)}{\Vert }^{2}I(T(\mathbb{X})\le T_0/n^{1/p})\right\}$$

Full text

KEY WORDS: Convex optimizations, Curvature regularization, Nonparametric regression, Piecewise linear, Prediction.

1 Introduction

In recent years, the great success of the deep neural network (DNN) has ignited the machine learning community in formulating predictive models as solutions to differentiable optimization problems. The distributions of samples, e.g., images and languages, are assumed to be concentrated in the regions of some low-dimensional smooth functionals (manifolds), such that even if the samples are not strictly on the manifold, the error can be very small. More specifically, the manifold assumption can be summarized in three folds:

• (1)

The underlying model (i.e., the principle functional) for the data distribution is on a low-dimensional functional.

• (2)

The principal functional is smooth but complicated in shape.

• (3)

The errors between the the functional and real data are very small relative to the variance of the functional.

These assumptions are the prerequisites of the success of DNN. First, the DNN is a universal approximator, which ensures its flexibility in shape when approximating the principal surface of the model. Second, the sparse techniques used in DNN, e.g., dropout and pooling, make it possible to regularize the network to concentrate on low-dimensional manifolds. Third, the sample size used in the training process of DNN is usually large, which ensures that DNN can well approximate the principle functional. Therefore, DNN is a successful example of a nonparametric, flexible and differentiable approximator, yet with its only weakness on interpretability. We focus on the problem of approximating a low-dimensional smooth functional with small errors or without errors, and propose an interpretable, geometrically optimal, nonparametric and differentiable approach, the Delaunay triangulation learner (DTL).

The DTL has a number of attractive properties:

• (1)

DTL fits a piecewise linear model, which is well established and can be easily interpreted.

• (2)

DTL naturally separates the feature space in a geometrically optimal way, and it has local adaptivity in each separated region when fitting a smooth functional.

• (3)

Compared with the general triangulation methods (e.g., random triangulation), where the geometric structure of the triangles can be difficult to analyze, the Delaunay triangulation makes it possible for many stunning geometrical and statistical properties, which lays out a new direction for statistical research on piecewise linear models.

• (4)

Based on the construction of the DTL, we can define the regularization function geometrically to penalize the roughness of the fitted function.

• (5)

DTL can accommodate multidimensional subspace interactions in a flexible way, which is useful when the output of the model is dependent on the impact of a group of covariates.

• (6)

DTL is formulated as a differentiable optimization problem, which can be solved via a series of well-known gradient-based optimizers.

To better illustrate the advantages of the DTL both theoretically and empirically, we present the low-dimensional settings where there are only a small number of features ($p\ll n$), while leaving extensions to high-dimensional settings in our future work.

2 Delaunay Triangulation

For ease of exposition, we use ‘triangle’ and ‘simplex’ exchangeably throughout this paper, as triangle is a two-dimensional simplex. All definitions and theories are established on general dimension ($p\geq 2$). In geometry, triangulation is often used to determine the location of a point by forming simplex to the point from other known points. Let $\mathbb{P}$ be a set of $n$ points in the $p$-dimensional space ($n\geq p+1$), and the convex hull of the points has a nonzero volume. The Delaunay triangulation of the points is defined as follows.

Definition 2.1**.**

Given a set of points $\mathbb{P}$ in the $p$-Euclidean space, the Delaunay triangulation $DT(\mathbb{P})$ is a triangulation such that no point in $\mathbb{P}$ is inside the circumscribed spheres of other triangles in $DT(\mathbb{P})$.

This definition is illustrated in Figure 1 (a). Geometrically, it has been well established that for any given points $\mathbb{P}$ in general position (the convex hull of $\mathbb{P}$ is non-degenerate in $p$-dimensional space), there exists only one Delaunay triangulation $DT(\mathbb{P})$, i.e., the Delaunay triangulation is unique (Delaunay, 1934). Intuitively, the Delaunay triangulation algorithm results in a group of simplices that are most regularized in shape, in comparison to any other type of triangulation. As shown in Figure 1 (b) and (c), in a two-dimensional space, the mesh constructed by the Delaunay triangulation has fewer skinny triangles compared with that by a random triangulation. Figure 2 shows a saddle surface and its Delaunay triangulation.

Several algorithms can be used to implement the Delaunay triangulation, including the flip algorithm, the Bowyer–Watson algorithm (Bowyer, 1981), and (Watson, 1981), and the divide-and-conquer paradigm (Cignoni et al., 1998). The flip algorithm is fairly straightforward as it constructs the triangulation of points and flips the edges until every triangle is of the Delaunay type. It takes $O(n^{2})$ time for the edge-flipping operation, where $n$ is the number of points. The Bowyer–Watson algorithm is relatively more efficient as it takes $O(n\log(n))$ time, by repeatedly adding one vertex at a time and re-triangulating the affected parts of the graph. In the divide-and-conquer algorithm, a line is drawn recursively to split the vertices into two groups, and the Delaunay triangulation is then implemented for each group. The computational time can be reduced to $O(n\log\log n)$ owing to some tuning techniques. For higher dimensional cases ($p>2$), Fortune et al. (1992) proved that the plane-sweep approach produces the worst case time complexity of $O(n^{[p/2]})$ (Klee2015). In this paper, the Delaunay triangulation algorithm is implemented by using the Python SciPy package (Jones et al., 2001).

3 Delaunay Triangulation Learner

3.1 Formalization of DTL

As shown in Figure 2, given the data $\left\{\mathcal{X},\mathcal{Y}\right\}$ from a probabilistic model (joint distribution) $\mathcal{P_{X,Y}}$, where $\mathcal{Y}=\left\{y_{i}\right\}_{i=1}^{n}$ and $\mathcal{X}=\left\{\mathbb{x}_{i}\right\}_{i=1}^{n}$, the DTL is formalized as follows.

• (1)

Delaunay Partition

We partition the feature space with a triangle mesh by conducting Delaunay triangulation on $\mathcal{X}=\left\{\mathbb{x}_{i}\right\}_{i=1}^{n}$ and obtain a set of simplices, denoted as $\mathcal{D}\left(\mathcal{X}\right)$. The convex hull of the Delaunay triangulation, denoted as $\mathcal{H}\left(\mathcal{X}\right)$, is unique by definition. For any point $\mathbb{x}$ in the feature space, if $\mathbb{x}\in\mathcal{H}\left(\mathcal{X}\right)$, there exists a simplex in $\mathcal{D}\left(\mathcal{X}\right)$, denoted as $\mathcal{S}(\mathbb{x},\mathcal{X})$, that contains $\mathbb{x}$,

[TABLE]

where $I\left(\mathbb{x}\in\mathcal{S}\right)$ is the indicator function. If $\mathbb{x}\notin\mathcal{H}(\mathcal{X})$, there exists a point $\mathbb{x}(\mathcal{X})$ nearest to $\mathbb{x}$, with the corresponding response $y(\mathbb{x}(\mathcal{X})).$

• (2)

Parabolic Lifting

Let $\mathbb{\Psi}=\left(\psi_{1},\ldots,\psi_{n}\right)^{\top}$ denote a vector of location parameters. We construct the DTL as a linear interpolation function based on $\left\{\mathcal{X},\mathbb{\Psi}\right\}$, which is denoted as $F_{D}(\mathbb{x};\mathbb{\Psi})$. The linear interpolation function takes the form

[TABLE]

where $g(\mathbb{x};\mathcal{S}(\mathbb{x},\mathcal{X}),\mathbb{\Psi})$ is a $p$-dimensional linear function defined on the simplex $\mathcal{S}(\mathbb{x},\mathcal{X})$, satisfying

[TABLE]

In summary, the DTL partitions the feature space with a Delaunay triangulation algorithm. It is a nonparametric functional learner as the dimension of the parameter $\mathbb{\Psi}$ grows at the same pace as the sample size $n$.

3.2 Optimization

In the DTL functional optimization problem, one has a response variable $y$ and a vector of input variables $\mathbb{x}=\left(x_{1},\ldots,x_{p}\right)^{\top}$ in a $p$-dimensional feature space. Given the data $\left\{\mathcal{X},\mathcal{Y}\right\}$, our goal is to obtain an estimator $\hat{F}_{D}$ by solving an optimization problem,

[TABLE]

where $L(y,F_{D}(\mathbb{x};\mathbb{\Psi}))$ is the loss function, $\lambda$ is a tuning parameter and $R(F_{D}(\mathbb{x};\mathbb{\Psi}))$ is a regularization function measuring the roughness of $F_{D}(\mathbb{x};\mathbb{\Psi})$. The loss function can be squared loss or absolute loss for regression problems, and exponential loss for classification problems.

As for the regularization function $R(\cdot)$, we propose using the total discrete curvature as a measure of roughness. Suppose that a vertex ${\bf v}$ is inside the convex hull $\mathcal{H}(\mathcal{X})$, and let $\mathcal{N}({\bf v})$ denote the set of simplices that contain ${\bf v}$. As a linear interpolation function, the DTL is strictly linear in each triangulated region. Through the function $F_{D}(\mathbb{x};\mathbb{\Psi})$, each simplex $\mathcal{S}\in\mathcal{N}({\bf v})$ has a corresponding $p$-dimensional simplex $\mathcal{S}^{F}$ as its functional image. More specifically, $\mathcal{S}^{F}=\{(\mathbb{x},F_{D}(\mathcal{\mathbb{x}},\mathbb{\Psi}))|\mathbb{x}\in\mathcal{S}\}$ is a $p$-dimensional simplex for any $\mathcal{S}$. Furthermore, we define a $(p+1)$-dimensional vector $\mathbb{n}(\mathcal{S})$ as the standard up-norm vector of $\mathcal{S}^{F}$, which satisfies

[TABLE]

where ${\bf e}_{p+1}=(\mathbb{0}_{p},1)^{\top}$, $\mathbb{n}(\mathcal{S})[1:p]$ is the vector composed of the first $p$-dimensional elements of $\mathbb{n}(\mathcal{S})$, and $\mathbb{G}=\mathbb{\Delta}_{\mathbb{x}_{i}}^{-1}\Delta_{\psi_{i}}$ is a $p$-dimensional gradient vector of $\mathcal{S}^{F}$, where

[TABLE]

Here, $\mathbb{x}_{1},\ldots,\mathbb{x}_{p+1}$ are the vertices of the same simplex and $\psi_{1},\ldots,\psi_{p+1}$ are the corresponding location parameters of the DTL. The first equation in (3.1) ensures that the vector $\mathbb{n}(\mathcal{S})$ is orthogonal to the simplex $\mathcal{S}^{F}$. The second implies that the vector is normalized, and the third requires the vector to be in the upward direction.

As the up-norm vector shares the same projection as the gradient vector on the feature space, it takes the form of $\mathbb{n}(\mathcal{S})=\frac{(-\mathbb{G}^{\top},c)^{\top}}{\|(-\mathbb{G}^{\top},c)^{\top}\|}$, where $c$ is an unknown constant. By solving the equations in (3.1), we obtain $c=1$, and thus the standard up-norm vector $\mathbb{n}(\mathcal{S})$ can be written as

[TABLE]

Figure 3 exhibits the geometric view of $\mathbb{n}(\mathcal{S})$, and Figure 4 shows a plot of the standard up-norm vectors on a discrete surface.

Let $|\mathcal{N}({\bf v})|$ denote the number of elements in $\mathcal{N}({\bf v})$, and define the total discrete curvature as

[TABLE]

where $\measuredangle\left(\mathbb{n}(\mathcal{S}_{k}),\mathbb{n}(\mathcal{S}_{j})\right)$ represents the degree of the angle between $\mathbb{n}(\mathcal{S}_{k})$ and $\mathbb{n}(\mathcal{S}_{j})$, and $|\mathcal{N}({\bf v})|\choose 2$ is the number of all possible combinations for $\left(\mathbb{n}(\mathcal{S}_{k}),\mathbb{n}(\mathcal{S}_{j})\right)$. The total discrete curvature is calculated by measuring the angle between each pair of the up-norm vectors and averaging the total degree of the angles grouped by each vertex. A special case is that all the simplices of the DTL collectively form a hyperplane in the $(p+1)$-dimension, where all the standard up-norm vectors are identical and the total discrete curvature equals zero. As the cosine of an angle formed by two unit vectors is equal to the inner product of the vectors, it is equivalent to defining the regularization function as

[TABLE]

Both the loss function and the regularization function are differentiable with respect to $\mathbb{\Psi}$, thus all the gradient-based optimizers can be applied to solve the optimization problem of the DTL. The gradient of the objective function $L\left(y,F_{D}(\mathbb{x};\mathbb{\Psi})\right)+\lambda R\left(F_{D}(\mathbb{x};\mathbb{\Psi})\right)$ is given by $\nabla=\sum_{i=1}^{n}\frac{\partial L}{\partial\psi_{i}}+\lambda\sum_{i=1}^{n}\frac{\partial R}{\partial\psi_{i}}.$ Algorithm 1 displays the Adaptive Moment Estimation (Adam) algorithm (Kingma and Ba, 2015) that iteratively updates the vector $\mathbb{\Psi}$, where $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\varepsilon=10^{-8}$.

3.3 DTL Advantages

Figure 5 shows a comparison between the DTL, decision tree, and multivariate adaptive regression splines (MARS), fitted on the same data generated from a saddle surface model, $y=\arctan(x_{1}+x_{2})+\epsilon$, where the noise $\epsilon$ follows the normal distribution $N(0,0.01)$. In comparison with the decision tree and MARS, the DTL appears to be smoother and more flexible in shape. The tree-based method tends to be rough when estimating smooth functionals, as it is not easy to smooth out by fitting more complicated models in the terminal nodes. The MARS show better adaptivity in handling smooth nonlinear functions than decision trees. The DTL provides an intrinsic way to partition the feature space with the Delaunay triangulation algorithm, and it has the advantage of balancing the roughness and smoothness depending on the local availability of data and that can lead to better predictive accuracy. Figure 6 shows the behavior of the regularized DTL with different values of $\lambda$ when fitting a two-dimensional linear model under a squared loss function.

4 Theoretical Properties

We first study the geometrical optimality and asymptotic properties of the Delaunay triangulation mesh under general random distributions, and then these properties are used as cornerstones to find the statistical properties of the DTL.

4.1 Geometrical Optimality Properties

We define a geometric loss function to analyze the geometric optimality of the Delaunay triangulation with random points.

Definition 4.1**.**

Let $\mathcal{X}=\{\mathbb{X}_{1},\ldots,\mathbb{X}_{n}\}$ $(n>p+1)$ be $n$ points in $R^{p}$ in general position with a convex hull $\mathcal{H}(\mathcal{X})$, and $\mathcal{T}$ be any triangulation of $\mathcal{X}$. For any point $\mathbb{X}\in R^{p}$, if $\mathbb{X}\in\mathcal{H}(\mathcal{X})$, the geometric loss function of the points is defined as

[TABLE]

where $\mathbb{X}_{(i)},i=1,\ldots,p+1$ are the vertices of the triangle in $\mathcal{T}$ that covers $\mathbb{X}$. Otherwise, if $\mathbb{X}\notin\mathcal{H}(\mathcal{X})$, we define $H(\mathbb{X},\mathcal{X},\mathcal{T})=0.$

Theorem 1**.**

Let $\mathcal{X}=\{\mathbb{X}_{1},\ldots,\mathbb{X}_{n}\}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. Then,

[TABLE]

where $\mathcal{D}$ is the Delaunay triangulation of $\mathcal{X}$.

Proof.

For any point $\mathbb{X}\in R^{p}$ and a simplex with vertices $\mathcal{X}_{p+1}=\{\mathbb{X}_{1},\ldots,\mathbb{X}_{p+1}\},$ if $\mathbb{X}\in\mathcal{H}(\mathcal{X}_{p+1})$, define the generalized geometric loss function as

[TABLE]

Otherwise, if $\mathbb{X}\notin\mathcal{H}(\mathcal{X}_{p+1}),$ $\tilde{H}(\mathbb{X},\mathcal{X}_{p+1})=0.$ By Theorem 1 in Rajan (1991), among all the triangles with vertices in $\mathcal{X}$ that contain the point $\mathbb{X}$, the Delaunay triangle minimizes the geometric loss function $\tilde{H}(\mathbb{X},\mathcal{X}_{p+1})$ of the triangle at the point. Thus, for any triangulation $\mathcal{T}$, we have

[TABLE]

Thus,

[TABLE]

∎

4.2 Geometrical Asymptotic Properties

Since the DTL makes prediction for a point by fitting a linear interpolation function locally on the Delaunay triangle that covers the point, the asymptotic behavior of the covering triangle is crucial to the asymptotics of DTL in both regression and classification problems. We show the convergence rate of the size of the covering triangle in terms of its average edge length.

Lemma 1**.**

Let $\mathbb{X},\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ take values on $[0,1]^{p}$. Let $\mathbb{X}_{(1)}$ denote the nearest neighbour of $\mathbb{X}$ among $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$. Then, for $p\geq 2$,

[TABLE]

where $c_{p}=\frac{4(1+\sqrt{p})^{2}}{V_{p}^{2/p}}$ and $V_{p}$ is the volume of a unit $p$-ball; and for $p=1$, it reduces to

[TABLE]

Proof.

Let $\mathbb{X}_{(i,1)}$ denote the nearest neighbor of $\mathbb{X}_{i}$, and let $r_{i}=\|\mathbb{X}_{(i,1)}-\mathbb{X}_{i}\|.$ Define the $p$-ball $B_{i}=\left\{\mathbb{x}\in\mathbb{R}^{p}:\|\mathbb{x}-\mathbb{X}_{i}\|<r_{i}/2\right\}.$ Note that the $p$-balls are disjoint, and $r_{i}\leq\sqrt{p}$, and we have that $\cup_{i=1}^{n+1}B_{i}\subseteq[-\frac{\sqrt{p}}{2},1+\frac{\sqrt{p}}{2}]^{p}$. Thus,

[TABLE]

where $\mu$ is the Lebesgue measure. Hence,

[TABLE]

For $p\geq 2$, it can be shown that

[TABLE]

which proves the case of $p\geq 2$. For $p=1$, it is obvious that

[TABLE]

∎

Lemma 2**.**

Let $\mathbb{X},\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. Let $H_{n}$ denote the event that $\mathbb{X}$ falls inside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$. Then, $\mathbb{P}(H_{n})\rightarrow 1$, as $n\rightarrow\infty$.

Proof.

Let $U_{j}(\delta),j=1,\ldots,2^{p}$, denote a series of $\delta$-square-neighborhoods of the vertices of the $[0,1]^{p}$, and $\mathcal{X}=\{\mathbb{X}_{1},\ldots,\mathbb{X}_{n}\}.$ Define the event $E_{n}=\cap_{j=1}^{2^{p}}\{U_{j}\cap\mathcal{X}\neq\varnothing\}$. Then, given any arbitrary small value of $\delta>0$, we have

[TABLE]

which implies $\mathbb{P}({E_{n}})\rightarrow 1$, as $n\rightarrow\infty.$ Furthermore,

[TABLE]

which is guaranteed by the convexity of the convex hull. Thus, $\mathbb{P}(H_{n})\rightarrow 1,$ as $n\rightarrow\infty.$ ∎

Theorem 2**.**

Let $\mathbb{X},\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. If the point $\mathbb{X}$ is inside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, define $T(\mathbb{X})$ as the average length of the edges of the Delaunay triangle that covers $\mathbb{X}$; otherwise, $T(\mathbb{X})=\|\mathbb{X}-\mathbb{X}_{(1)}\|$, where $\mathbb{X}_{(1)}$ is the nearest neighbor point of $\mathbb{X}$ among $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$. Then,

[TABLE]

where $\beta$ is a positive constant depending only on the dimension $p$ and density $f$.

The proof of Theorem 2 contains two parts. Following the machinary of the 1-NN proof (Biau et al., 2015), the first part shows that the convergence rate of $T(\mathbb{X})$ for the out-of-hull case is $\mathcal{O}(n^{-1/p})$, and since the point $\mathbb{X}$ falls inside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ with probability one, the convergence rate is determined by the inside-hull case. The second part of the proof follows the results of Jimenez and Yukich (2002).

Proof of Theorem 2.

Let $H_{n}$ denote the event that point $\mathbb{X}$ falls inside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, and by the definition of $T(\mathbb{X})$ we have

[TABLE]

where $\tilde{T}(\mathbb{X})$ is the average length of the edges of the Delaunay triangle that covers point $\mathbb{X}$, and if $\mathbb{X}$ is outside the convex hull, define $\tilde{T}(\mathbb{X})=0.$

We first show $\mathbb{E}\left\{(1-I(H_{n}))\|\mathbb{X}-\mathbb{X}_{(1)}\|\right\}=o(n^{-1/p}).$ By adopting the techniques in Biau et al. (2015), we have

[TABLE]

where $B(\mathbb{X},\sqrt{t})$ is a $p$-ball with center $\mathbb{X}$ and radius $\sqrt{t}$, and $\mu$ is the probability measure. By the Lebesgue differentiation theorem, at Lebesgue-almost all $\mathbb{x}$,

[TABLE]

where $V_{p}$ is the volume of a standard $p$-ball. At such $\mathbb{x}$, we have for fixed $t$,

[TABLE]

Furthermore, Fatou’s lemma implies

[TABLE]

To establish the upper bound for $p>2$, we take an arbitrarily large constant $T_{0}$ and split

[TABLE]

By Fatou’s lemma, the first term $\mathbb{I}$ can be bounded as follows,

[TABLE]

Then, we show that the second term $\mathbb{II}$ is controlled by the choice of $T_{0}$. By the symmetry of the nearest neighbor distance, we can rewrite $\mathbb{II}$ as

[TABLE]

Denote $r_{i}=\|\mathbb{X}_{i}-\mathbb{X}_{(i,1)}\|$, and it has been proved in Lemma 1 that

[TABLE]

Set $K=\sum_{i=1}^{n+1}I(r_{i}>T_{0}/n^{1/p})$, and we know that $K\leq na_{p}/T_{0}^{p}$. By Jensen’s inequality, when $K>0$,

[TABLE]

Thus,

[TABLE]

As a result, we have shown for $p>2$,

[TABLE]

For $p=2$, (4.1) leads to

[TABLE]

As $n\mathbb{E}\|\mathbb{X}_{(1)}-\mathbb{X}\|^{2}=\frac{n}{n+1}\mathbb{E}(\sum_{i=1}^{n+1}r_{i}^{2}),$ we have $n\mathbb{E}\|\mathbb{X}_{(1)}-\mathbb{X}\|^{2}<\frac{4}{\pi}(1+\sqrt{2})^{2}.$ By the Cauchy–Schwarz inequality,

[TABLE]

As shown in Lemma 2, $\mathbb{P}(H_{n})\rightarrow 1$, as $n\rightarrow\infty$. Thus, $\mathbb{E}\{(1-I(H_{n}))\|\mathbb{X}-\mathbb{X}_{(1)}\|\}=o(n^{-1/p}).$

Next, we show that $n^{1/p}\mathbb{E}\{I(H_{n})\tilde{T}(\mathbb{X})\}\rightarrow\beta\int_{[0,1]^{p}}f(\mathbb{x})^{(p-1)/p}d\mathbb{x}.$ Let $N(n)$ denote the number of triangles in the Delaunay triangulation of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n},$ and let $\Delta_{1},\ldots,\Delta_{N}(n)$ denote the triangles. By definition, we show that $\mathbb{E}\tilde{T}(\mathbb{X})=\frac{1}{2N(n)}\sum_{i=1}^{N(n)}\sum_{e\in S(\mathbb{x}_{i})}|e|,$ where $S(\mathbb{x}_{i})$ is the set of edges in the Delaunay graph that are connected with point $\mathbb{x}_{i}$, and $|e|$ is the length of edge $e$. As shown in Lemma 2,

[TABLE]

as $n\rightarrow\infty.$ By the symmetry of triangle indices, we have

[TABLE]

Thus, we have $N(n)\mathbb{P}(\mathbb{X}\in\Delta_{1})\rightarrow 1.$ Hence, by reindexing the points of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n},$

[TABLE]

where $S(\mathbb{x}_{i})$ denotes the set of edges that are connected with point $\mathbb{x}_{i}.$

By Theorem 2.1 in Jimenez and Yukich (2002), it is shown that for the Delaunay graph,

[TABLE]

where $A_{i}$’s are the distances between the points $\mathbb{s}_{i},i=1,\ldots,m$, to the original point $\mathbb{0},$ and $\mathbb{s}_{i},i=1,\ldots,m$, are from a $p$-dimensional unit intensity Poisson process with density $f(\mathbb{x}).$

With $p=1$ and since $I(H_{n})\rightarrow 1$ in probability, we obtain

[TABLE]

where $\beta$ is a constant that depends only on dimension $p$ and density $f$. ∎

Corollary 1**.**

Let $\mathbb{X},\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. If the point $\mathbb{X}$ is inside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, define $\mathbb{X}_{(1)}$ as any one of the vertices of the covering Delaunay triangle; otherwise, define $\mathbb{X}_{(1)}$ as the nearest neighbor point of $\mathbb{X}.$ Then, $\mathbb{X}_{(1)}\rightarrow\mathbb{X}$ in probability.

Proof.

We need to show that $\|\mathbb{X}_{(1)}-\mathbb{X}\|\leq\tilde{T}(\mathbb{X}).$ If point $\mathbb{X}$ is outside the convex hull of $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, $\|\mathbb{X}_{(1)}-\mathbb{X}\|=\tilde{T}(\mathbb{X})$, by definition. Otherwise, $\|\mathbb{X}-\mathbb{X}_{(1)}\|\leq\max_{j}\|\mathbb{X}_{(1)}-\mathbb{X}_{(j)}\|\leq{{p+1}\choose{2}}\tilde{T}(\mathbb{X}).$ From Theorem 2, we conclude that $\mathbb{E}\|\mathbb{X}_{(1)}-\mathbb{X}\|\rightarrow 0$, as $n\rightarrow\infty,$ which implies that $\mathbb{X}_{(1)}\rightarrow\mathbb{X}$ in probability. ∎

This result shows that the vertices of the covering simplex of point $\mathbb{X}$ all converge to $\mathbb{X}$ in probability.

4.3 General Case

For regression problems, the DTL is shown to be consistent, and for classification problems, the DTL has an error rate smaller than or equal to $2R_{B}(1-R_{B})$, where $R_{B}$ is the Bayes error rate (the minimum error rate that can be achieved by any function approximator).

Regression

Theorem 3**.**

Assume that $\mathbb{E}y^{2}<\infty$, and $\psi(\mathbb{X})=\mathbb{E}\left(y|\mathbb{X}\right)$ is continuous. Then, the DTL regression function estimate $\hat{F}_{D}$ satisfies

[TABLE]

where $L^{*}=\inf_{g}\mathbb{E}|y-g(\mathbb{X})|^{2}$ is the minimal value of the $L_{2}$ risk over all continuous functions $g:\mathbb{R}^{p}\rightarrow\mathbb{R}$.

Proof.

If point $\mathbb{X}$ is outside the convex hull $\mathcal{H}_{n}$, $\hat{F}_{D}(\mathbb{X})=y_{(1)}$, where $y_{(1)}$ is the corresponding response of the nearest neighbor point $\mathbb{X}_{(1)}.$ If $\mathbb{X}$ is inside $\mathcal{H}_{n}$, denote the simplex covering $\mathbb{X}$ as $\Delta_{\mathbb{X}},$ and the vertices of the simplex $\Delta_{\mathbb{X}}$ as $\mathbb{x}(\Delta_{\mathbb{X}})_{[1]},\ldots,\mathbb{x}(\Delta_{\mathbb{X}})_{[p+1]},$ or simply $\mathbb{x}_{[1]},\ldots,\mathbb{x}_{[p+1]}.$ Thus, for any interior point $\mathbb{x}$ of simplex $\Delta_{\mathbb{X}},$ it can be represented as $\mathbb{x}=\sum_{j=1}^{p+1}\lambda_{[j]}(\mathbb{X})\mathbb{x}_{[j]},$ where $\lambda_{[j]}(\mathbb{X})>0$, for $j=1,\ldots,p+1$, and $\sum_{j=1}^{p+1}{\lambda_{[j]}(\mathbb{X})}=1.$ As the DTL is linear inside the simplex, and $\hat{F}_{D}(\mathbb{x})=\sum_{j=1}^{p+1}\lambda_{[j]}(\mathbb{X})y_{[j]},$ where $y_{[j]}$ is the response to corresponding point $\mathbb{x}_{[j]}.$ In general, $\hat{F}_{D}(\mathbb{X})=I_{n}\sum_{j=1}^{p+1}\lambda_{[j]}(\mathbb{X})y_{[j]}+(1-I_{n})y_{(1)},$ with $I_{n}=I(H_{n})$.

Thus, we have

[TABLE]

From Corollary 1, we know that $\mathbb{X}_{[j]}\rightarrow\mathbb{X}$ for each $j=1,\ldots,p+1$, in probability. By the continuity of function $\psi(\mathbb{x})$, the second term in (4.3) goes to zero, i.e., \mathbb{E}\big{|}I_{n}\sum_{j=1}^{p+1}\lambda_{[j]}(\mathbb{X})\psi(\mathbb{X}_{[j]})-\psi(\mathbb{X})\big{|}^{2}\rightarrow 0. For the third term in (4.3), the Cauchy–Schwarz inequality implies

[TABLE]

As $n\rightarrow\infty$, $\mathbb{E}(I_{n})\rightarrow 1$, then

[TABLE]

and \mathbb{E}\big{|}I_{n}\sum_{j=1}^{p+1}\lambda_{[j]}(\mathbb{X})\psi(\mathbb{X}_{[j]})-\psi(\mathbb{X})\big{|}^{2}=o(1).

For the first term in (4.3), we have

[TABLE]

and again by the continuity of $\psi(\mathbb{x})$,

[TABLE]

Since $L^{*}$ is the minimum value of $\mathbb{E}|\hat{F}_{D}(\mathbb{X})-\psi(\mathbb{X})|^{2},$ the consistency is shown. ∎

Classification

Theorem 4**.**

For a two-class classification model, if the conditional probability $\psi(\mathbb{x})=\mathbb{P}\left(y=1|\mathbb{x}\right)$ is a continuous function of $\mathbb{x}$. The mis-classification risk $R$ of a DTL classifier is bounded as

[TABLE]

where $R_{B}$ is the Bayes error of the model.

Proof.

By Lemma 2, the point $\mathbb{X}$ is covered by a Delaunay triangle with probability one. Denote the vertices of the covering triangle as $\mathbb{X}_{[1]},\ldots,\mathbb{X}_{[p+1]}.$ Define $r_{n}(\mathbb{X})$ as the empirical risk of the DTL classifier,

[TABLE]

where $\hat{y}=I(\hat{F}_{D}(\mathbb{X})>1/2)$ is the prediction of the DTL classifier, and $\sum_{j=1}^{p+1}\lambda_{[j]}=1.$ Since Corollary 1 shows that for any $j$, $\mathbb{X}_{[j]}\rightarrow\mathbb{X}$ in probability, based on the continuity of $\psi(\cdot)$, we have $\psi(\mathbb{X}_{[j]})\rightarrow\psi(\mathbb{X})$ in probability. Thus, $r_{n}(\mathbb{X})\rightarrow 2\psi(\mathbb{X})(1-\psi(\mathbb{X}))$, in probability. The conditional Bayes risk $r^{*}(\mathbb{X})=\min\{\psi(\mathbb{X}),1-\psi(\mathbb{X})\}.$ By the symmetry of function $r^{*}$ in $\psi(\cdot)$, we write

[TABLE]

By definition, the DTL risk $R=\lim_{n\rightarrow\infty}\mathbb{E}\{r_{n}(\mathbb{X})\},$ and since $r_{n}(\cdot)$ is bounded by 1, applying the dominated convergence theorem, $R=\mathbb{E}\{\lim_{n\rightarrow\infty}r_{n}(\mathbb{X})\}.$ The limit yields

[TABLE]

As the Bayes risk is the expectation of $r^{*}$, we have $R=2R_{B}(1-R_{B})-2{\rm Var}\{r^{*}(\mathbb{X})\},$ and thus, $R\leq 2R_{B}(1-R_{B}).$ ∎

4.4 Smooth and Noiseless Case

As we consider the local behavior at $\mathbb{x}$, without loss of generality, we assume $\mathbb{x}=\mathbb{0}$.

Lemma 3**.**

Assume that $\mathbb{0}$ is a Lebesgue point of $f$, $f(\mathbb{0})>0$. Denote $\mathbb{X}_{1},\ldots,\mathbb{X}_{p+1}$ as the vertices of the Delaunay triangle that covers the point $\mathbb{0}$, and further let $\mathbb{X}_{(1)},\ldots,\mathbb{X}_{(p+1)}$ denote these vertices ordered by their distances from $\mathbb{0}$. Then,

[TABLE]

where $G_{i}=\sum_{j=1}^{i}E_{j}$, and $E_{1},\ldots,E_{p+1}$ are independent standard exponential random variables, and $\mathbb{Z}_{1},\ldots,\mathbb{Z}_{p+1}$ are independent random vectors uniformly distributed on the surface of $B(\mathbb{0},1)$, a $p$-ball with center $\mathbb{0}$ and radius $1$.

Proof.

Let $K$ be a positive constant to be chosen later. Consider a density $g_{K,n}$ related to $f$ as follows: let

[TABLE]

and set

[TABLE]

which is a proper density (i.e., nonnegative and integrating to one) for $n$ large enough. Note that

[TABLE]

Therefore, by Doeblin’s coupling method, there exist random variables $\mathbb{X}$ and $\mathbb{Y}$ with corresponding densities $\mathbb{f}$ and $g_{K,n}$, such that $\mathbb{P}(\mathbb{Y}\neq\mathbb{X})=\frac{1}{2}\int_{\mathbb{R}^{p}}|g_{K,n}(\mathbb{x})-f(\mathbb{x})|d\mathbb{x}=o(\frac{1}{n}).$ ∎

Repeating this $n$ times, we create two coupled samples of random variables that are $i.i.d.$ within the sample. The sample $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ is drawn from the distribution of $\mathbb{X}$, and the sample $\mathbb{Y}_{1},\ldots,\mathbb{Y}_{n}$ is drawn from the distribution of $\mathbb{Y}$. Recall that the total variation distance between two random vectors $\mathbb{W}$, $\mathbb{W}^{{}^{\prime}}\in R^{n}$ is defined by $d_{\rm TV}(\mathbb{W},\mathbb{W}^{{}^{\prime}})=\sup_{A\in\mathcal{B}}|\mathbb{P}\{\mathbb{W}\in A\}-\mathbb{P}\{\mathbb{W}^{{}^{\prime}}\in A\}|,$ where $\mathcal{B}$ denotes the Borel sets of $\mathbb{R}^{p}.$ Let $\|\mathbb{Y}_{(1)}\|\leq\ldots\leq\|\mathbb{Y}_{(n)}\|$ and $\|\mathbb{X}_{(1)}\|\leq\ldots\leq\|\mathbb{X}_{(n)}\|$ be the reordered samples. Then

[TABLE]

Define

[TABLE]

where $\nu$ is the probability measure of $\mathbb{Y}$. We recall that $U_{(1)},\ldots,U_{(p+1)}$ are uniform order statistics, and thus $n(U_{(1)},\ldots,U_{(p+1)})\xrightarrow[]{\mathcal{D}}(G_{1},\ldots,G_{p+1}).$ In fact, this convergence is also in the $d_{\rm TV}$ sense. Also, if $\|Y_{(p+1)}\|\leq K/n^{1/p}$, then $U_{(i)}=f(\mathbb{0})V_{p}\|\mathbb{Y}_{(i)}\|^{p}.$ Thus,

[TABLE]

for all $K$ large enough, depending on $\epsilon.$ Let $\mathbb{Z}_{1},\ldots,\mathbb{Z}_{p+1}$ be $i.i.d.$ random vectors uniformly distributed on the surface of $B(\mathbb{0},1).$ Then,

[TABLE]

as on $\|\mathbb{Y}\|\leq K/n^{1/p},$ $\mathbb{Y}$ has a radially symmetric distribution. Therefore,

[TABLE]

which concludes the proof.

Theorem 5**.**

Assume that $\mathbb{0}$ is a Lebesgue point of the density $f$ and $f(\mathbb{0})>0$, and the regression function $r$ is continuously differentiable in a neighborhood of $\mathbb{0}$. Then,

• (1)

$(f(\mathbb{0})V_{p+1}n)^{1/p}\left(\frac{1}{p+1}\sum_{i=1}^{p+1}\sum_{j=1}^{p}r_{j}^{{}^{\prime}}(\mathbb{0})\mathbb{X}_{(i,j)}\right)\xrightarrow[]{\mathcal{D}}\frac{1}{p+1}\sum_{i=1}^{p+1}G_{i}^{1/p}\left(\sum_{j=1}^{p}r_{j}^{{}^{\prime}}(\mathbb{0})\mathbb{Z}_{i,j}\right).$**

• (2)

$r_{n}(\mathbb{0})-r(\mathbb{0})=O_{\mathbb{P}}\left(\frac{1}{\sqrt{p+1}}\left(\frac{p+1}{n}\right)^{1/p}\right).$**

Proof.

By Taylor’s series approximation,

[TABLE]

(where $\psi(\mathbb{x})=o(\|\mathbb{x}\|)$ as $\|\mathbb{x}\|\rightarrow 0)$. Observe that $\frac{\sum_{i=1}^{p+1}\psi(\mathbb{X}_{(i)})}{\|\mathbb{X}_{(p+1)}\|}\rightarrow 0$ in probability, if $\|\mathbb{X}_{(p+1)}\|\rightarrow 0$ in probability. But, by Lemma 3 $\|\mathbb{X}_{(p+1)}\|=O_{\mathbb{P}}((\frac{p+1}{n})^{1/p})$, and therefore,

[TABLE]

Still, by Lemma 3,

[TABLE]

Define $L_{p+1}=\sum_{i=1}^{p+1}\lambda_{i}G_{i}^{1/p}\mathbb{Z}_{i,1},$ we have

[TABLE]

Thus, since $\mathbb{E}\mathbb{Z}_{1,1}^{2}=\frac{\|\mathbb{Z}_{1}\|^{2}}{p}=1/p,$

[TABLE]

which implies that

[TABLE]

∎

Theorem 6**.**

Let $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. Then, for a function $y=\phi(\mathbb{x})$ which has an upper-bounded second-order derivative, we have

[TABLE]

where $\hat{F}_{D}$ is a DTL estimate of $\phi$ based on samples $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, $\|\nabla^{2}\phi\|$ is the upper bound of the second-order derivative, $\beta$ is a positive constant depending only on the dimension $p$ and density $f$, and $\|\cdot\|$ is a $L_{2}$ norm of the continuous functional space.

Proof.

Define $\mathcal{D}$ as the Delaunay triangulation on $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$, and $\mathcal{S}_{1},\ldots,\mathcal{S}_{N_{\mathcal{D}}}$ as the Delaunay simplices, where $N_{\mathcal{D}}$ is the number of simplices in $\mathcal{D}$. Define $T_{j}$ as the average length of the edges of the Delaunay triangle $\mathcal{S}_{j}$. By the $L_{2}$-error bound obtained in the inequality (3.22) in Waldron (1998), for each simplex $\mathcal{S}_{j}$, we have

[TABLE]

where $\|\nabla_{j}^{2}\phi\|=\||\sup_{\mathbb{x}\in\mathcal{S}_{j}}\nabla^{2}(\mathbb{x})\phi|\|$ is the supremum of the second-order derivative of $\phi$ on simplex $\mathcal{S}_{j},$ and $h_{j}$ is the maximum edge of $\mathcal{S}_{j}$. Then, it can be shown that

[TABLE]

where $\|\nabla^{2}\phi\|=\max_{j}\|\nabla_{j}^{2}\phi\|<\infty$. Thus,

[TABLE]

and by Theorem 2, we have

[TABLE]

∎

Theorem 7** (Local Adaptivity).**

Let $\mathbb{X}_{1},\ldots,\mathbb{X}_{n}$ be i.i.d samples from a continuous density $f$, which is bounded away from zero and infinity on $[0,1]^{p}$. Assume that the regression function $r$ is continuously differentiable in $[0,1]^{p}$. Then, for any point $\mathbb{x}\in[0,1]^{p}$, $\hat{F}_{D}^{\prime}(\mathbb{x})\rightarrow r^{\prime}(\mathbb{x})$, as $n\rightarrow\infty,$ where $\hat{F}_{D}^{\prime}(\mathbb{x})$ is the gradient of the DTL at point $\mathbb{x}.$

Proof.

For any fixed point $\mathbb{x}$, by Taylor series approximation,

[TABLE]

(where $\psi(\mathbb{x})=o(\|\mathbb{x}\|)$ as $\|\mathbb{x}\|\rightarrow 0)$. Observe that $\frac{\sum_{i=1}^{p+1}\psi(\mathbb{X}_{(i)})}{\|\mathbb{X}_{(p+1)}\|}\rightarrow 0$ in probability, if $\|\mathbb{X}_{(p+1)}\|\rightarrow 0$ in probability. Based on Theorem 2 and Corollary 1, we have $\mathbb{X}_{(i)}\rightarrow\mathbb{X}$, $i=1,\ldots,p+1.$ ∎

Based on the multivariate mean-value theorem, there exists a point $\xi\in\mathcal{S}(\mathbb{x})$ such that $\hat{F}_{D}^{\prime}(\mathbb{x})=r^{\prime}(\xi).$ As $n\rightarrow\infty$, $\lim_{n\rightarrow\infty}\hat{F}_{D}^{\prime}(\mathbb{x})=\lim_{n\rightarrow\infty}r^{\prime}(\xi_{n})=r^{\prime}(\lim_{n\rightarrow\infty}\xi_{n})=r^{\prime}(\mathbb{x}).$

4.5 Regularized Estimates

Under regularizations, the estimated parameter vector is given by

[TABLE]

To see how accurate the regularized estimate is, we compare $\hat{\mathbb{\Psi}}_{\lambda}$ with the population optimal solution $\mathbb{\Psi}^{*}=\operatorname*{arginf}_{\mathbb{\Psi}}\mathbb{E}_{\mathcal{P_{\mathcal{X},\mathcal{Y}}}}L(\mathcal{Y},F_{D}(\mathcal{X};\mathbb{\Psi})).$

Definition 4.2**.**

Given the location parameter vector $\mathbb{\Psi}\in R^{n}$, the regularization function $R$ is said to be decomposable with respect to $(\mathcal{M},\bar{\mathcal{M}}^{\perp})$, if a pair of subspaces $\mathcal{M}\subseteq\bar{\mathcal{M}}$ of $R^{n}$, $R(\mathbb{u}+\mathbb{v})=R(\mathbb{u})+R(\mathbb{v})$ for all $\mathbb{u}\in\mathcal{M}$ and $\mathbb{v}\in\bar{\mathcal{M}}^{\perp}$, where $\bar{\mathcal{M}}^{\perp}=\{\mathbb{v}\in R^{n}|\langle\mathbb{u},\mathbb{v}\rangle=0,\forall\mathbb{u}\in\bar{\mathcal{M}}\}.$

Definition 4.3**.**

The loss function satisfies a restricted strong convexity condition (RSCC) with curvature $\kappa_{L}>0$, if the optimal solution $\mathbb{\Psi}^{*}\in\mathcal{\mathcal{M}}$, $\delta L(\Delta,\mathbb{\Psi}^{*})\geq\kappa_{L}\|\Delta\|^{2}$ for all $\Delta\in\mathbb{C}(\mathcal{\mathcal{M}},\bar{\mathcal{\mathcal{M}}}^{\perp};\mathbb{\Psi}^{*})$, where $\mathbb{C}(\mathcal{M},\bar{\mathcal{M}}^{\perp};\mathbb{\Psi}^{*})=\{\Delta\in R^{n}|R(\Delta_{\bar{\mathcal{M}}^{\perp}})\leq 3R(\Delta_{\bar{\mathcal{M}}})+4R(\mathbb{\Psi}_{\mathcal{M}^{\perp}}^{*})\}$.

Theorem 8**.**

Based on a strictly positive regularization constant $\lambda\geq 2R(\nabla L(\mathbb{\Psi}^{*}))$, if the loss function satisfies RSCC, then we have $\|\hat{\mathbb{\Psi}}_{\lambda}-\mathbb{\Psi}^{*}\|\leq 9\frac{\lambda^{2}}{\kappa_{L}}\mathbb{\Phi}^{2}(\bar{\mathcal{M}})$ and $R(\hat{\mathbb{\Psi}}_{\lambda}-\mathbb{\Psi}^{*})\leq 12\frac{\lambda}{\kappa_{L}}\mathbb{\Phi}^{2}(\bar{\mathcal{M}}),$ where $\mathbb{\Phi}(\bar{\mathcal{M}})=\sup_{\mathbb{u}\in\mathcal{M}\{\mathbb{0}\}}\frac{R(\mathbb{u})}{\|\mathbb{u}\|}.$

Proof.

We first show that the regularization function of DTL is decomposable, and then we use Theorem 1 and Corollary 1 in Negahban (2012) to prove the theorem. For a given set of parameters $\mathcal{M}$, define $\mathcal{H}(\mathcal{M})$ as its convex hull, and $\bar{\mathcal{M}}$ as the set of points that are connected to any point of $\mathcal{H}(\mathcal{M})$ by an edge of the Delaunay triangulation $\mathcal{D}(\mathcal{X})$. Then, it is clear that based on the definition of geometric regularization function, we have $R(\mathbb{u}+\mathbb{v})=R(\mathbb{u})+R(\mathbb{v})$ for all $\mathbb{u}\in\mathcal{M}$ and $\mathbb{v}\in\bar{\mathcal{M}}^{\perp}$. Thus if the loss function satisfies RSCC for such an $\mathcal{M}$, then we can make the conclusion.

∎

5 Comparisons of Statistical Learners

We use two artificial classification datasets to visualize how the DTL-based methods differ from the neural network and tree-based methods. In particular, we demonstrate the smoothness and robustness of the DTL-based methods, when handling feature interaction problems.

The artificial datasets are generated from three models of the Python Scikit learn package. The ‘moons’ model has two clusters of points with moon shapes, and the ‘circles’ model consists of two clusters of points distributed along two circles with different radiuses. As shown in Figures 7–8, the data are displayed with blue points for $y=1$ and red points for $y=0$. Three classifiers are considered including the DTL, neural network, decision tree, and the estimated probability function of each classifier is plotted with color maps.

We first illustrate the local adaptivity of the DTL. As shown in panel (a) of Figures 7–8, the estimated probability functions of the DTL are piecewise linear in the convex hull of the observations. This feature provides the DTL the advantage when approximating smooth boundary, since it can locally choose to use a piecewise linear model to estimate the probability function. Figure 7 (b) exhibits a less adaptive classification probability (either 0 or 1) using neural network, and Figure 7 (c) shows a clear stairwise shape of the classification region using the decision tree.

We next point out that the DTL can more reliably capture the feature interactions. As exhibited in Figure 7 (c) and Figure 8 (c), there are some spiky regions in the estimated probability functions using the decision tree. It shows the weakness of the decision tree in capturing the feature interactions via the marginal tree splitting approach.

6 Discussion

We have presented the DTL as a differentiable and nonparametric algorithm that has simple geometric interpretations. The geometrical and statistical properties of the DTL are investigated to explain its advantages under various settings, including general regression, classification models, smooth and noiseless models. These properties signify the importance of developing more applications based on this piecewise linear learner, e.g., the DTL can be used as an alternative approach to substituting the nonlinear activation function in the neural network. The DTL can also be generalized into manifold learning approaches, with multi-dimensional input and output. There exist a series of open questions warranting further investigations. In terms of theory, we have focused on low-dimensional settings ($p<n$) for simplicity, while it would be more interesting to generalize the DTL into high-dimensional settings. As for computation, investigation on GPU-based parallel computing for DTL is also warranted.