Guarantees on Nearest-Neighbor Condensation heuristics

TL;DR

This paper introduces theoretical guarantees for nearest-neighbor condensation algorithms, proposing two new methods with proven bounds and comparing them to existing algorithms to understand their reduction capabilities.

Contribution

It presents the first theoretical bounds for practical NN condensation algorithms and introduces two new algorithms, RSS and VSS, with provable size guarantees.

Findings

RSS and VSS have provable upper bounds on selected subset size

Comparison of MSS and FCNN with new algorithms reveals differences in reduction efficiency

Theoretical analysis provides insights into the selection size of state-of-the-art algorithms

Abstract

The problem of nearest-neighbor (NN) condensation aims to reduce the size of a training set of a nearest-neighbor classifier while maintaining its classification accuracy. Although many condensation techniques have been proposed, few bounds have been proved on the amount of reduction achieved. In this paper, we present one of the first theoretical results for practical NN condensation algorithms. We propose two condensation algorithms, called RSS and VSS, along with provable upper-bounds on the size of their selected subsets. Additionally, we shed light on the selection size of two other state-of-the-art algorithms, called MSS and FCNN, and compare them to the new algorithms.

Full text

Guarantees on Nearest-Neighbor Condensation heuristics††thanks: Research supported by NSF grant CCF–1618866.

Alejandro Flores-Velazco University of Maryland, College Park, afloresv@cs.umd.edu   

David Mount University of Maryland, College Park, mount@umd.edu

Abstract  The problem of nearest-neighbor (NN) condensation aims to reduce the size of a training set of a nearest-neighbor classifier while maintaining its classification accuracy. Although many condensation techniques have been proposed, few bounds have been proved on the amount of reduction achieved. In this paper, we present one of the first theoretical results for practical NN condensation algorithms. We propose two condensation algorithms, called RSS and VSS, along with provable upper-bounds on the size of their selected subsets. Additionally, we shed light on the selection size of two other state-of-the-art algorithms, called MSS and FCNN, and compare them to the new algorithms.

1 Introduction

In machine learning, classification involves a training set $P\subset\mathbb{R}^{d}$ of $n$ labeled points in Euclidean space. The label $l(p)$ of each point $p\in P$ indicates the class to which the point belongs to, partitioning of $P$ into a finite set of classes. Given an unlabeled query point $q\in\mathbb{R}^{d}$ the goal of a classifier is to predict $q$’s label using $P$.

The nearest-neighbor (NN) rule is among the best-known classification techniques [6]. It classifies a query point $q$ with the label of its closest point in $P$, according to some metric. Throughout, we will assume the Euclidean $\ell_{2}$ metric. Despite its simplicity, the NN rule exhibits good classification accuracy both experimentally and theoretically  [13, 4, 5]. However, it’s often criticized due to its high space and time complexities. This raises the question of whether it is possible to replace $P$ with a significantly smaller subset without affecting the classification accuracy under the NN rule. This problem is called nearest-neighbor condensation. In this paper we propose two new NN condensation algorithms and analyze their worst-case performance.

1.1 Preliminaries

For any point $p\in P$, define an enemy of $p$ to be any point in $P$ of different class than $p$. The nearest enemy (NE) of $p$, denoted $\textup{ne}(p)$, is the closest such point, and its distance from $p$, called the NE distance, is denoted as $\textup{{d}}_{\textup{ne}}(p)=\textup{{d}}(p,\textup{ne}(p))$. Similarly, denote the NN distance as $\textup{{d}}_{\textup{nn}}(p)=\textup{{d}}(p,\textup{nn}(p))$. Define the NE ball of $p$ to be the ball centered at $p$ with radius $\textup{{d}}_{\textup{ne}}(p)$. Let $\kappa$ denote the number of distinct NE points of $P$.

A point $p\in P$ is called a border point if it is incident to an edge of the Delaunay triangulation of $P$ whose opposite endpoint is an enemy of $p$. Otherwise, $p$ is called an internal point. By definition, the border points of $P$ completely characterize the portion of the Voronoi diagram that separates Voronoi cells of different classes. Let $k$ denote the number of border points of $P$.

1.2 Related work

A subset $R\subseteq P$ is said to be consistent if for all $p\in P$ its nearest neighbor in $R$ is of the same class as $p$. Intuitively, $R$ is consistent if and only if every point of $P$ is correctly classified using the NN rule over $R$. Formally, nearest-neighbor condensation involves finding an (ideally small) consistent subset of $P$ [9].

Other criteria for condensation have been studied in the literature. One such criterion is known as selectivity [12]. A subset $R\subseteq P$ is said to be selective if and only if for all $p\in P$, its nearest neighbor in $R$ is closer to $p$ than its nearest enemy in $P$. Clearly selectivity implies consistency, as the NE distance in $R$ of any point of $P$ is at least its NE distance in $P$. Note that neither consistency or selectivity imply that every query point of $\mathbb{R}^{d}$ is correctly classified, just those in $P$.

The strongest criteria, known as Voronoi condensation, consists of selecting all border points of $P$ [16]. This guarantees the correct classification of any query point in $\mathbb{R}^{d}$. In contrast, a consistent subset only guarantees correct classification of $P$. For the case when $P\subset\mathbb{R}^{2}$, an output-sensitive algorithm was proposed [3] for finding all border points of $P$ in $\mathcal{O}(n\log{k})$ worst-case time. Unfortunately, it is not known how to generalize this algorithm to higher dimensions, and a straightforward approach suffers from the super-linear worst-case size of the Delaunay triangulation.

In general, it has been shown that the problems of computing consistent and selective subsets of minimum cardinality are both NP-complete [17, 18, 11]. Thus, most research has focused on practical heuristics. For comprehensive surveys, see [14, 15, 10]. CNN (Condensed Nearest-Neighbor) [9] was the first algorithm proposed for computing consistent subsets. Even though it has been widely cited in the literature, CNN suffers from several drawbacks: its running time is cubic in the worst-case, and the resulting subset is order-dependent, meaning that the result is determined by the order in which points are considered by the algorithm. Alternatives include FCNN (Fast CNN) [1] and MSS (Modified Selective Subset) [2], which produce consistent and selective subsets respectively. Both algorithms run in $\mathcal{O}(n^{2})$ worst-case time, and are order-independent. These algorithms are considered the state-of-the-art for the NN condensation problem, subject to achieving these properties. While such heuristics have been extensively studied experimentally [10, 7], theoretical results are scarce. Unfortunately, to the best of our knowledge, no bounds are known for the size of the subsets generated by any of these heuristics.

More recently, an approximation algorithm called NET [8] was proposed, along with almost matching hardness lower bounds for the problem. The idea is to compute a $\gamma$-net of $P$, with $\gamma$ equal to the minimum NE distance in $P$, implying that the resulting subset is consistent. Unfortunately, while NET has provable worst-case performance, this approach allows little room for condensation, and in practice, the resulting subset can be too large. For example, on the training set in Figure 1(a), NET selects a subset of over 90% of the points, while other algorithms select only 3% of the points.

1.3 Our contributions

In this paper, we propose two new NN condensation algorithms, called RSS and VSS. We will establish asymptotically tight upper-bounds on the sizes of their selected subsets. Moreover, we prove that these algorithms have similar complexity to the popular state-of-the-art algorithms FCNN and MSS. Additionally, we also analyze the selection size of FCNN and MSS. To the best of our knowledge, these represent the first theoretical results on practical NN condensation algorithms. The following is a summary of our contributions.

[TABLE]

2 Results on condensation size

One of the most significant shortcomings in research on practical NN condensation algorithms is the lack of theoretical results on the sizes of the selected subsets. Typically, the performance of these heuristics has been established experimentally.

We establish bounds with respect to the size of two well-known and structured subsets of points: (a) the set of all NE points of $P$ of size $\kappa$, and (b) the set of border points of $P$ of size $k$. Throughout the paper, we refer equally to the algorithms and their selected subsets.

2.1 The state-of-the-art

Let’s begin our analysis with a state-of-the-art algorithm for the problem: MSS or Modified Selective Subset (see Algorithm 1). The selection process of the algorithm can be simply described as follows: for every $p\in P$, MSS selects the point with smaller NE distance contained inside the NE ball of $p$.

Clearly, this approach computes a selective subset of $P$, which by definition, is order-independent. MSS can be implemented in $\mathcal{O}(n^{2})$ worst-case time. Unfortunately, the selection criteria of MSS can be too strict, requiring one particular point to be added for each point $p\in P$. Note that any point inside the NE ball of $p$ suffices for achieving selectiveness. In practice, this can lead to much larger subsets than needed.

This intuition is formalized in the following theorem. Here we show that the subset selected by MSS can select a subset of unbounded size as a function of $\kappa$ or $k$.

Theorem 2.1**.**

Given $0<\varepsilon<1$, there exists a training set $P\subset\mathbb{R}^{d}$ with a constant number of NE and border points such that MSS selects $\Omega(1/\varepsilon)$ points.

Proof 2.2**.**

Recall that for each point in $P$, the MSS algorithm selects the point inside its NE ball with minimum NE distance. Given a parameter $0<\varepsilon<1$, we construct a training set in $d$-dimensional Euclidean space, as illustrated in Figure 2(a).

Create two points $r_{1}$ and $r_{2}$, and assign them to the class of red points. Without lost of generality, the distance between these two points is 1. Let $\vec{u}$ be the unit vector from $r_{1}$ to $r_{2}$, create additional points $b_{i}=r_{1}+\frac{i\varepsilon}{4}\vec{u}$ for $i=\{1,2,\dots,3/\varepsilon\}$. Assign all $b_{i}$ points to the class of blue points. The set of all these points constitute the training set $P$. It is easy to prove that $P$ has only 4 NE points and 4 border points, corresponding to $r_{1},r_{2},b_{1}$ and $b_{3/\varepsilon}$.

Let’s discuss which points are added by MSS for each point in $P$ (see Figure 2(b)). For points $r_{1}$ and $r_{2}$, the only points inside their NE balls are themselves, so both $r_{1}$ and $r_{2}$ belong to the subset selected by MSS. For points $b_{i}$ with $i\leq 2/\varepsilon$, the point with minimum NE distance contained inside their NE ball is $b_{1}$, which is also added to the subset. Now, consider the points $b_{i}$ with $2/\varepsilon<i<5/2\varepsilon$. Let $j=i-2/\varepsilon$, it is easy to prove that the point with minimum NE distance inside the NE ball of $b_{i}$ is $b_{2j+1}$ (see Figure 2(b)). Therefore, this implies that the number of points selected by MSS equals $5/2\varepsilon-2/\varepsilon=1/2\varepsilon=\Omega(1/\varepsilon)$.

2.2 A better approach

We propose a new algorithm, called RSS or Relaxed Selective Subset, with the idea relaxing the selection process of MSS, while still computing a selective subset. For a given point $p\in P$, while MSS requires to add the point with smallest NE distance inside the NE ball of $p$, in RSS any point inside the NE ball $p$ suffices.

The idea is rather simple (see Algorithm 2). Points of $P$ are examined in increasing order with respect to their NE distance, and we add any point whose NE ball contains no point previously added by the algorithm. This tends to select points close to the decision boundary of $P$ (see Figure 1(d)), as points far from the boundary are examined later in the selection process, and are more likely to already contain points inside their NE ball.

Theorem 2.3**.**

RSS* is order-independent and computes a selective subset of $P$ in $\mathcal{O}(n^{2})$ worst-case time.*

Proof 2.4**.**

By construction, every point in $P$ was either added into RSS, or has a point in RSS inside its NE ball. Therefore, RSS is selective. The order-independence follows from the initial sorting step.

Let’s analyze the time complexity of RSS. The initial step requires $\mathcal{O}(n^{2})$ time for computing the NE distances of each point in $P$, plus additional $\mathcal{O}(n\log{n})$ time for sorting the points according to such distances. The main loop iterates through each point in $P$, and searches their nearest neighbor in the current subset, incurring into additional $\mathcal{O}(n^{2})$ time. Finally, the worst-case time complexity of the algorithm is $\mathcal{O}(n^{2})$.

Theorem 2.5**.**

RSS* selects at most $\mathcal{O}(\kappa\ (3/\pi)^{d-1})$ points.*

Proof 2.6**.**

The proof follows by a charging argument on each NE point of $P$. Consider a NE point $p\in P$, and let $\textup{RSS}_{p}$ be the set of points selected by RSS such that $p$ is their NE. Let $p_{i},p_{j}\in\textup{RSS}_{p}$ be two such points, and without loss of generality say that $\textup{{d}}_{\textup{ne}}(p_{i})\leq\textup{{d}}_{\textup{ne}}(p_{j})$. By construction of the algorithm, we also know that $\textup{{d}}(p_{i},p_{j})\geq\textup{{d}}_{\textup{ne}}(p_{j})$. Now, consider the triangle $\triangle pp_{i}p_{j}$. Clearly, the side $pp_{i}$ is the larger of such triangle, and therefore, the angle $\angle p_{i}pp_{j}\geq\pi/3$. Meaning that the angle between any two points in $\textup{RSS}_{p}$ with respect to $p$ is at least $\pi/3$.

By a standard packing argument, this implies that $|\textup{RSS}_{p}|=\mathcal{O}((3/\pi)^{d-1})$. Finally, we obtain that $|\textup{RSS}|=\sum_{p}|\textup{RSS}_{p}|=\kappa\ \mathcal{O}((3/\pi)^{d-1})$.

For constant dimension $d$, the size of RSS is $\mathcal{O}(\kappa)$. Therefore, the following result implies that the upper-bound on RSS is tight up to constant factors. Furthermore, it implies that this is the best upper-bound we can hope to achieve in terms of $\kappa$.

Theorem 2.7** (Lower-bound).**

There exists a training set $P\subset\mathbb{R}^{d}$ with $\kappa$ NE points, for which any consistent subset contains $\Omega(\kappa\kern 1.0ptc^{d-1})$ points, for some constant $c$.

Proof 2.8**.**

We construct a training set $P$ in $d$-dimensional Euclidean space, which contains points of two classes: red and blue. Consider the following arrangement of points: create a red point $p$, and take every point at distance 1 from $p$ as a blue point. Simply, the points on the surface of a unit ball centered at $p$.

Take any consistent subset of this training set, and consider some point $p^{\prime}$ in such subset, and the bisector between $p$ and $p^{\prime}$. The intersection between this bisector and the unit ball centered at $p$ describes a cap of such ball of height $1/2$. Any point located inside this cap is closer to $p^{\prime}$ than $p$, and hence, correctly classified. Clearly, by definition of consistency, all points in the ball must be covered by at least one cap. By a simple packing argument, we know such covering needs $\Omega(c^{d-1})$ points, for some constant $c$.

So far the training set constructed has only two nearest enemy points; the red point $p$, and one blue point closest to $p$ (assuming general position). Then, we can repeat this arrangement $\kappa/2$ times, using sufficiently separated center points. This generates a training set $P$ with a number of NE points equal to $\kappa$, for which any consistent subset has size $\Omega(\kappa\kern 1.0ptc^{d-1})$.

Different parameters from $\kappa$ can be used to bound the selection size of condensation algorithms. Let’s consider $k$, the number of border points in the training set $P$. From the example illustrated in Figure 3, we know that RSS can select more points than $k$ (see Figure 3(b)). Repeating such arrangement forces RSS to select $\Omega(k)$ points. Yet, the question remains, at most, how many more points than $k$ can this algorithm select?

Lemma 2.9**.**

The nearest enemy point of any point in $P$ is a also a border point of $P$.

Proof 2.10**.**

Take any point $p\in P$. Consider the empty ball of maximum radius, tangent to point $\textup{ne}(p)$, and with center in the line segment between $p$ and $\textup{ne}(p)$. Being maximal, this ball is tangent to another point $p^{*}\in P$ (see Figure 4(a)). Clearly, $p^{*}$ is inside the NE ball of $p$, which implies that $p$ and $p^{*}$ belong to the same class, making $p^{*}$ and $\textup{ne}(p)$ enemies. By the empty ball property, this means that both $p^{*}$ and $\textup{ne}(p)$ are border points of $P$.

From Lemma 2.9, we know that in Euclidean space, the number of NE points of $P$ is at most the number of border points of $P$. That is, $\kappa\leq k$. While this implies an easy extension of the bound for RSS, now in terms of $k$, it is unclear if the other factors can be improved.

Alternatively, this opens another idea for condensation. In order to prove Lemma 2.9, we show that there exist at least one border point inside the NE ball of any point $p\in P$. Therefore, any algorithm that only selects such border points, can guarantee to compute a selective subset of size at most $k$. Consider then a modification of RSS, where for each point $p_{i}\in P$, if no other point lying inside the NE ball of $p$ has been added yet, instead of adding $p_{i}$ as RSS does, we add a border point inside NE ball of $p$. We call this new algorithm VSS or Voronoi Selective Subset (see Algorithm 3).

Theorem 2.11**.**

VSS* computes a selective subset of $P$ of size at most $k$ in $\mathcal{O}(n^{2})$ worst-case time.*

Proof 2.12**.**

By construction, for any point in $p\in P\setminus\textup{VSS}$ the algorithm selected one border point inside the NE ball of $p$. This implies that the resulting subset is selective, and contains no more than $k$ points.

Now, we describe an efficient implementation of VSS. Recall that for every point $p\in P\setminus\textup{VSS}$, the algorithm finds a border point inside its NE ball. Without loss of generality implement VSS to compute the point $p^{*}$ that minimizes the radius of an empty ball tangent to both $\textup{ne}(p)$ and $p^{*}$, and center in the line segment between $p$ and $\textup{ne}(p)$. For any given point $p^{\prime}$ inside the NE ball of $p$, denote $r(p,p^{\prime})$ to be the radius of the ball tangent to $p^{\prime}$ and $\textup{ne}(p)$ and center in the line segment between $p$ and $\textup{ne}(p)$. As illustrated in Figure 4(b), let vectors $\vec{u}=\frac{p-\textup{ne}(p)}{\|p-\textup{ne}(p)\|}$ and $\vec{v}=p^{\prime}-\textup{ne}(p)$, the radius of this ball can be derived from the formula $r(p,p^{\prime})=\|\vec{v}+r(p,p^{\prime})\vec{u}\|$ as $r(p,p^{\prime})=\frac{\vec{v}\cdot\vec{v}}{2\vec{u}\cdot\vec{v}}$.

As $p^{*}$ is defined as the point that minimizes such radius, a simple scan over the points of $P$ suffices to identify the corresponding $p^{*}$ for any point $p\in P$. Therefore, this implies that VSS can be computed in $\mathcal{O}(n^{2})$ worst-case time.

2.3 What about FCNN?

FCNN or Fast Condensed Nearest-Neighbor is yet another popular state-of-the-art algorithm for the NN condensation problem. In contrast with MSS, which finds selective subsets, FCNN selects consistent subsets of $P$.

Let’s now describe the selection process of FCNN (see Algorithm 4). Essentially, FCNN maintains a subset of $P$, which is updated in each iteration, by adding points that are incorrectly classified using the current subset. The iterations stop when all points of $P$ are correctly classified by the current subset, that is, when FCNN is consistent. Starting with the centroids of each class, set $S$ contains some misclassified points from $P\setminus\textup{FCNN}$ that will be added in the next iterartion. How does the algorithm decide which points to include in $S$? Define $\textup{voren}(p,\textup{FCNN},P)$ as the set of enemy points of $p$ in $P$, whose NN in FCNN is $p$, that is, the set $\{q\in P\mid l(q)\neq l(p)\wedge\textup{ne}(q,\textup{FCNN})=p\}$. Then, for each point $p\in\textup{FCNN}$, the algorithm selects one representative out of its corresponding $\textup{voren}(p,\textup{FCNN},P)$, which is usually defined as the NN to $p$.

Theorem 2.13**.**

There exists a training set $P\subset\mathbb{R}^{d}$ with $k$ border points, for which FCNN selects $\Omega(k)$ points.

Proof 2.14**.**

Consider the arrangement in Figure 5(b) (left), consisting of points of 4 classes. The centroids of the blue, yellow, and red classes are the only points labeled as such. By placing a sufficient number of black points far at the top of this arrangement, we avoid their centroid to be any of the three black points in the arrangement. Beginning with the centroids, the first iteration of FCNN would have added the points outlined in Figure 5(b) (right). Now each of these points have one black point inside their Voronoi cells, and therefore, these black points will be the representatives added in the second iteration. This small example, with $k=5$, shows how to force FCNN to add all the border points plus two internal points. Out of these two internal black points, one is the centroid added in the initial step. The remaining internal black point, however, was added by the algorithm during the iterative process. This scheme can be extended to larger values of $k$, without increasing the number of classes.

The previous is the first building block of the entire training set, shown in Figure 5(a). To this “middle” arrangement, we append “side” arrangements of points, as the one illustrated in Figure 5(c), which will have similar behavior to the middle arrangement. This particular side arrangement will be appended to the right of the middle one, such that the distance between the red points is greater than the distance from the yellow to the red point. Every time we append a new side arrangement, its blue and red labels are swapped. The arrangements appended to the left side are simply a horizontal flip of the right arrangement. Now, the behavior of FCNN in such a setup is illustrated with the arrows in Figure 5(a). The extreme point of the previous arrangement adds the yellow point at the center of the current arrangement, which then adds the red point next to the blue point, as is closer than the other red point. Next, this red point adds the blue point, and the yellow point adds the remaining red point. Finally, the Voronoi cells of these points will look as shown in Figure 5(c) (right), and in the next iteration, the tree black points will be added.

After adding side arrangements as needed (same number of the left and right), it is easy to show that the centroids are still the tree points in the middle arrangement and the black point at the top (by adding a sufficient number of black points in the top cluster). This implies than FCNN will be forced to select $\Omega(k)$ points.

While this example sheds light on the selection behavior of FCNN, an upper-bound is still missing. Based on the following lemma, we conjecture that FCNN selects at most $\mathcal{O}(\kappa\ c^{d-1})$ points, for some constant $c$.

Lemma 2.15**.**

Consider a point $p$ selected by FCNN. Then, the number of representatives of $p$ selected throughout the algorithm does not exceed $\mathcal{O}((3/\pi)^{d-1})$ points.

Proof 2.16**.**

This proof follows from similar arguments to the ones described in the proof of Theorem 2.5. Consider $p_{1},p_{2}\in\textup{FCNN}$ to be two points added to by the algorithm as representatives of $p$. Without loss of generality, $p_{1}$ was added before $p_{2}$, implying that $\textup{{d}}(p,p_{1})\leq\textup{{d}}(p,p_{2})$. By construction, if $p_{2}$ was added as a representative of $p$, and not of $p_{1}$, we also know that $\textup{{d}}(p,p_{2})\leq\textup{{d}}(p_{1},p_{2})$. From this, consider the triangle $\triangle pp_{1}p_{2}$ and the angle $\angle p_{1}pp_{2}$. This is the largest angle of the triangle, meaning that $\angle p_{1}pp_{2}\geq\pi/3$. Finally, by a standard packing argument, there are at most $\mathcal{O}((3/\pi)^{d-1})$ such points.

3 Open problems

A few key questions remain unsolved:

• •

In terms of $k$, our best upper-bound on the selection size of RSS is not tight. Can it be improved?

• •

Is it possible to prove an upper-bound on the selection size of FCNN in terms of either $\kappa$ or $k$?