Kernel Treelets

TL;DR

Kernel Treelets (KT) is a novel hierarchical clustering method that integrates multiscale data decomposition with kernel functions, enabling application to non-numeric data and providing multi-resolution analysis.

Contribution

It introduces Kernel Treelets, combining treelets with kernel methods, allowing hierarchical clustering on general data types in feature space.

Findings

KT effectively clusters non-numeric data.

KT produces multi-resolution basis sequences.

Examples demonstrate KT's effectiveness in clustering.

Abstract

A new method for hierarchical clustering is presented. It combines treelets, a particular multiscale decomposition of data, with a projection on a reproducing kernel Hilbert space. The proposed approach, called kernel treelets (KT), effectively substitutes the correlation coefficient matrix used in treelets with a symmetric, positive semi-definite matrix efficiently constructed from a kernel function. Unlike most clustering methods, which require data sets to be numeric, KT can be applied to more general data and yield a multi-resolution sequence of basis on the data directly in feature space. The effectiveness and potential of KT in clustering analysis is illustrated with some examples.

Figures

Tables

Table 1. Table 1: Method and Runtime Table for Figure 1

Method\Dataset	1	2	3	4	5	6
0 - KTrbf	2.003	2.063	2.325	2.094	2.819	1.967
1 - KTlinear	1.585	1.613	1.402	1.73	2.341	1.469
2 - KTpoly	3.956	6.08	6.878	9.582	9.836	4.526
3 - MiniBatchKMeans	0.006	0.018	0.009	0.01	0.007	0.009
4 - MeanShift	0.047	0.032	0.063	0.057	0.032	0.05
5 - SpectralClustering	0.642	1.011	0.13	0.352	0.257	0.208
6 - Ward	0.114	0.098	0.513	0.245	0.111	0.087
7 - AgglomerateClustering	0.085	0.102	0.374	0.196	0.103	0.078
8 - DBSCAN	0.015	0.014	0.015	0.012	0.067	0.012
9 - GaussianMixture	0.005	0.005	0.008	0.012	0.004	0.009

Table 2. Table 2: Method and Runtime Table for Figure 2

Method\Dataset	1	2	3	4	5	6
0 - KT50	0.011	0.012	0.013	0.011	0.011	0.01
1 - KT100	0.035	0.044	0.039	0.045	0.033	0.028
2 - KT200	0.109	0.099	0.128	0.12	0.121	0.132
3 - KT300	0.225	0.217	0.242	0.269	0.259	0.235
4 - KT500	0.551	0.568	0.62	0.569	0.652	0.536
5 - KT800	1.315	1.513	1.534	1.378	1.699	1.295
6 - KT1000	2.016	2.055	2.336	2.098	2.782	1.941
7 - KT1200	2.88	2.94	3.242	3.004	4.146	2.77
8 - KT1499	4.438	4.532	5.4	4.713	6.788	4.341
9 - KT1500	4.472	4.69	5.398	4.807	6.782	4.274

Equations

$$[J^{T}MJ{]}_{pq}=[J^{T}MJ{]}_{qp}=0.$$

$$\left[\begin{array}{cc}c& -s\\ s& c\end{array}\right]\left[\begin{array}{cc}{M}_{pp}& M_{pq}\\ M_{qp}& M_{qq}\end{array}\right]\left[\begin{array}{cc}c& s\\ -s& c\end{array}\right]=\left[\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right]$$

$$b=\frac{M_{pp}-M_{qq}}{2M_{pq}}.$$

$$t=\frac{sgn(b)}{|b|+\sqrt{b^2+1}}.$$

$$[M_k{]}_{ij}=\sqrt{\frac{[A_{k-1}{]}_{ij}^2}{[A_{k-1}{]}_{ii}[A_{k-1}{]}_{jj}}}+\lambda |[A_{k-1}{]}_{ij}|.$$

$${\alpha }_k,{\beta }_k=\underset{\alpha ,\beta \in S_{k-1}}{\mathrm{argmax}}[M_k{]}_{\alpha \beta }.$$

$$B_k=J_k^T{J}_{k-1}^T\cdots J_2^T{J}_1^T,$$

$$A_k=B_k{A}_0{B}_k^T.$$

$${\tau }_k(v)=B_{k}v-\sum _{\begin{array}{c}i\notin S_i\\ |B_{k}v\cdot e_i|<ϵ\end{array}}(B_{k}v\cdot e_i)e_i.$$

$$K(x_1,x_2)$$

$$\sum _{i=1}^s\sum _{j=1}^s{c}_i{c}_{j}K(x_i,x_j)$$

$$K(x_1,x_2)=⟨{\Phi }_K(x),{\Phi }_K(y){⟩}_{\mathbb{H}}.$$

$$K(x_1,x_2)=\exp \{-\frac{||x_1-x_2|{|}^2}{2{\sigma }^2}\}.$$

$$K(x_1,x_2)=(\alpha ⟨x_1,x_2⟩+c_0{)}^r.$$

$$\Vert x_1-x_2{\Vert }^2=⟨x_1-x_2,x_1-x_2⟩=⟨x_1,x_1⟩+⟨x_2,x_2⟩-2⟨x_1,x_2⟩.$$

$$\Vert x_1-x_2{\Vert }^2$$

$$=K(x_1,x_1)+K(x_2,x_2)-2K(x_1,x_2).$$

$$d(v_1,v_2{)}^2=K(v_1,v_1)+K(v_2,v_2)-2K(v_1,v_2)$$

$$[{\Phi }_K(d_i){]}_B=\left[\begin{array}{c}{\delta }_i\\ 0\end{array}\right],$$

$$\eta =\text{dim}(\text{span}\{{\Phi }_K(d_i):i=1,2,...,n\})\le n.$$

$$[{\Phi }_K(d_i){]}_{\hat{B}}=\left[\begin{array}{c}{\hat{d}}_i\\ 0\end{array}\right].$$

$$\left[\begin{array}{cccc}{\hat{d}}_1& {\hat{d}}_2& \cdots & {\hat{d}}_n\end{array}\right]=U\Sigma V^T.$$

$$[{\Phi }_K(d_i){]}_B=\left[\begin{array}{cc}V{U}^T& 0\\ 0& I\end{array}\right][{\Phi }_K(d_i){]}_{\hat{B}}=\left[\begin{array}{c}V{U}^T{\hat{d}}_i\\ 0\end{array}\right]=\left[\begin{array}{c}{\delta }_i\\ 0\end{array}\right].$$

$$\left[\begin{array}{cccc}{\delta }_1& {\delta }_2& \cdots & {\delta }_n\end{array}\right]=Q{U}^T\left[\begin{array}{cccc}{\hat{d}}_1& {\hat{d}}_2& \cdots & {\hat{d}}_n\end{array}\right]=Q\Sigma Q^T$$

$$\left[\begin{array}{c}\Psi (v)\\ \ast \end{array}\right]=[{\Phi }_K(v){]}_B.$$

$$\left[\begin{array}{c}\Psi (d_i)\\ 0\end{array}\right]=[{\Phi }_K(d_i){]}_B,$$

$$⟨\Psi (D),\Psi (D)⟩={\left[\begin{array}{cccc}{\delta }_1& {\delta }_2& \cdots & {\delta }_n\end{array}\right]}^2=⟨{\Phi }_K(D),{\Phi }_K(D){⟩}_{\mathbb{H}}.$$

$$Cov(\Psi (D))=\Psi (D)\Psi (D{)}^T=\Psi (D{)}^2=⟨\Psi (D),\Psi (D)⟩=⟨{\varphi }_K(D),{\varphi }_K(D){⟩}_{\mathbb{H}}=K(D,D).$$

$$A_0=Cov(\Psi (D))=K(D,D).$$

$$K(v_1,v_2)=\{\begin{array}{ll}1045& v_1=v_2\\ 1& v_1,v_{2}areconnected\\ 0& otherwise\end{array}$$

$$\sum _{j\ne i}|K(V,V{)}_{i,j}|=deg(v_i)\le \underset{\zeta }{\max }deg(v_{\zeta })=1045=K(V,V{)}_{i,i}.$$

Full text

1 Introduction

Treelets, introduced by Lee, Nadler, and Wasserman [1, 2], is a method to produce a multiscale, hierarchical decomposition of unordered data. The central premise of Treelets is to exploit sparsity and capture intrinsic localized structures with only a few features, represented in terms of an orthonormal basis. The hierarchical tree constructed by the treelet algorithm provides a scale-based partition of the data that can be used for classification, specially for cluster analysis [3].

Cluster analysis, also called clustering, is concerned with finding a partition of a set such that its corresponding equivalence class captures similarity of its elements. The Treelet approach is an example of hierarchical clustering (HC) [4], which is a type of methods that provides a nested and multiscale clustering. The typical complexity of HC methods is $O(n^{3})$ (where $n$ denotes the number of data in the dataset) but Treelets, like single linkage HC [5] and complete linkage HC [6], can be done in $O(n^{2})$ operations. Most of these clustering methods are only applicable to numerical dataset only. However, many modern datasets do not have clear representations in $\mathbb{R}^{p}$ due for example to missing data, length difference, and non-numeric attributes. A typical solution to this problem usually involves finding a projection from each observation to $\mathbb{R}^{p}$ as is the case for example in text vectorization [7], array alignment [8], and missing-data imputation [9]. These particular projections pose considerable challenges and might raise the bias of the model if false assumptions are made.

In this paper we propose a HC method that combines Treelets with a projection on a feature space that is a Reproducing Kernel Hilbert Space (RKHS). We call this method Kernel Treelets (KT). It effectively substitutes the correlation coefficient matrix, used by the original treelet method as a measure of similarity among variables, with a symmetric, positive semi-definite matrix constructed from a (Mercer) kernel function. The intuition behind this approach is that inner products provide a measure of similarity and a projection into a RKHS, done via the so-called Kernel trick [10, 11], is a natural and efficient way to construct appropriate similarity matrices for a wide variety of data sets, including those mentioned above. We present some examples that demonstrate the potential of KT as an effective tool for clustering analysis.

2 Background Information

We provide in this section a brief description of the Treelet algorithm [1, 2] and the Kernel method [12]. Treelets are based on the repeated application of two dimensional (Jacobi) rotations to a matrix measuring the similarity of variables. So we start by reviewing Jacobi (also called Givens) rotations first.

2.1 Jacobi Rotations

A Jacobi rotation matrix $J$ is an orthogonal matrix with at most 4 entries different from the identity, or more generally, a rotation operator on a 2 dimensional subspace generated by two coordinate axes. For a given symmetric matrix $M$ and entry $pq$ and rotation matrix $J$ is constructed so that

[TABLE]

The construction of $J$ is equivalent to finding the cosine (c) and sine (c) of the angle of rotation, which satisfy

[TABLE]

subject to the constraint $c^{2}+s^{2}=1$. The matrix $J$ is then given

• •

$J_{pp}=J_{qq}=c$

• •

$J_{pq}=-J_{qp}=s$

• •

For other entries $ij$, $J_{ij}=I_{ij}$.

A numerical stable way of computing this problem is as follows:

• •

Assume $M_{pq}\neq 0$, and compute

[TABLE]

• •

Let $sgn(b)$ be 1 if $b\geq 0$ and -1 otherwise, then we define

[TABLE]

• •

From which we can calculate $c=\frac{1}{\sqrt{t^{2}+1}}$ and $s=ct$.

The complexity of storing a Given’s rotation matrix is $O(1)$, and Jacobi rotation over a $n\times n$ matrix uses $O(1)$ space with time complexity $O(n)$.

2.2 Treelets

The Treelets algorithm [1, 2] was designed to construct a multiscale basis and a corresponding hierarchical clustering over the attributes of some datasets in $\mathbb{R}^{p}$, to exploit sparsity. In its most efficient implementation [2] it is an $O(np^{2})$ algorithm. The algorithm starts with a regularization, hyper-parameter $\lambda$ and computing a $p\times p$ (empirical) covariance matrix $A_{0}$. The initial scaling indices are defined as the set $S_{0}=\{1,2,...p\}$. With base case $A_{0}$ and $S_{0}$, each step $A_{k}$ and $S_{k}$ for $k\in\{1,2,3,...,p-1\}$ can be constructed inductively as follows:

Construct matrix $M_{k}$ of the same shape as $A_{0}$ entry-wise:

[TABLE] 2. 2.

Find the two indices $\alpha_{k},\beta_{k}$ such that

[TABLE]

. 3. 3.

Calculate Jacobi rotation matrix $J_{k}$ for $\alpha_{k},\beta_{k}$ and matrix $A_{k}=J_{k}^{T}A_{k-1}J_{k}$. 4. 4.

Without loss of generality, $\alpha_{k}$ and $\beta_{k}$ is interchangeable, so we require that $[A_{k}]_{\alpha_{k}\alpha_{k}}\leq[A_{k}]_{\beta_{k}\beta_{k}}$, and record $\alpha_{k}$ and $\beta_{k}$. 5. 5.

Define $S_{k}=S_{k-1}-\{\alpha_{k}\}$.

2.2.1 Treelets Transform and Treelets Basis

The Jacobi rotations produce a Treelets basis for each $k\in\{1,2,3,...,p-1\}$. The sequence of matrices $\{J_{k}\}$ provides a basis for $\mathbb{R}^{p}$, defined as

[TABLE]

such that

[TABLE]

So for every vector $v\in\mathbb{R}^{p}$, there is a $k$th basis representation $B_{k}v$. Furthermore, there is a compressed $k$th basis representation obtained by dropping insignificant ($<\epsilon$) non-scaling indices of $B_{k}v$. That is, if we define $e_{i}$ to be the $i$th column of the identity matrix, the compressed $k$th basis representation is given by

[TABLE]

2.2.2 Treelets Hierarchical Clustering

Treelets is also a hierarchical clustering method over the attributes. The hierarchical clustering structure is stored in $\alpha_{k},\beta_{k}$. We start with trivial clustering where each element is in its own cluster and labeled by itself. For each $k$, we merge clusters labeled $\alpha_{k}$ and $\beta_{k}$ and label it $\beta_{k}$. This is feasible because each step $k$ the set of all cluster labels is exactly $S_{k-1}$. This operation gives a hierarchical tree for clustering use on the attributes.

2.3 Kernel Method

The Kernel method [12] allow us to map variables into a new feature space via a kernel function. We now review briefly the basic concepts and ideas of this approach (see for example [11]).

A kernel over some set $X$ is defined as a function $K:X\times X\to\mathbb{R}$. A symmetric and positive semi-definite (SPSD) kernel $K$ has the properties:

[TABLE]

If $X$ is finite, then $K$ is SPSD if and only if $K(X,X)$ is a SPSD matrix. If $X\subseteq\mathbb{R}^{p}$, then $K$ is SPSD if and only if there exists a function $\Phi_{K}:\mathbb{R}^{p}\to\mathbb{H}$, where $\mathbb{H}$ denotes the Hilbert space, such that for all $x_{1},x_{2}\in X$,

[TABLE]

The space $\mathbb{H}$ here is called a reproducing kernel Hilbert space (RKHS). The following are two common examples of SPSD kernels:

Radial basis function (RBF) kernel

[TABLE] 2. 2.

Polynomial kernel

[TABLE]

A kernel $K$ for a set $X$ can be restricted to a subset $Y\subseteq X$, and SPSD property is preserved during restriction. If the task is clustering over a finite set, the selected kernel needs only be SPSD on the set of all samples, which is generally finite, and we only need to check that the kernel matrix is SPSD. If we need to extend the clustering outcome to other data, e.g. clustering boosted classification, then $X$ has to include the whole data space as a subset.

2.4 K Nearest Neighbors (KNN)

K-nearest neighbors algorithm is a multi-class classification algorithm [13]. By specifying $k\in\mathbb{N}$ and a metric, the algorithm can, given a test data, predict its labels by the majority vote of a subset of $k$ closest elements in distance metric from training data. If an inner product is specified instead of distance, we can compute the distance between two point in the following way:

[TABLE]

If the metric is kernelized,

[TABLE]

2.5 Kernel Support Vector Machine (SVM)

Support Vector Machine (SVM) is a classification method by finding optimal hyper-planes. Kernel SVM [14] is a classification method towards nonlinear problems that performs SVM in RKHS generated by the kernel. When we only apply KT to a small sample, we may use kernel SVM with the same kernel to assign labels for data outside of this sample. This can be viewed as clustering attributes with treelets and using SVM to assign labels to other attributes in RKHS.

3 The KT Model

The task of KT is to find a clustering for some set $X$ given a SPSD kernel $K:X\times X\to\mathbb{R}$ measuring the similarity among variables. We combine Treelets with kernels by replacing the covariance $A_{0}$ with kernel matrix, and apply the rest of the steps of Treelets algorithm. The exact steps are as follows:

First we draw a sample $S$ with size $n_{S}$ from uniform distribution on $X$ and some sample size $n_{X}$. If more information about $X$ is given, it may be possible to draw a sample $S$ that better represent $X$ with smaller sample size. 2. 2.

Then, we calculate the kernel matrix $A_{0}=K(S,S)$. $A_{0}$ is a SPSD matrix because $K$ is SPSD, and thus we can apply Treelets algorithm with hyper-parameter $\lambda$ using $A_{0}$ instead of the (empirical) covariance matrix. $\lambda$ can be set to 0 or tuned experimentally as in Treelets. In this step, theTreelets method provides a hierarchical clustering tree of each columns of $A_{0}$, which corresponds to each observation in $S$. 3. 3.

If $S=X$, we are finished on the step above. Otherwise, we need to cluster the elements in $X$ based on clusters we have from elements in $S$. We use kernel SVM to complete this task. Given $S$ and its corresponding cluster labels, we train the kernel SVM with the same kernel $K$, and then apply to predict the cluster labels of $X$. K-Nearest Neighbors with distance induced by kernel

[TABLE]

is an alternative to kernel SVM.

3.1 Theory

We now prove that the kernel projection is equivalent to working with a symmetric positive definite matrix defined by the inner product in $\mathbb{H}$ and evaluated through the kernel. We also suggest a definition of a clustering setting and clustering equivalence that allows us to connect the results of the clustering analysis for the original set with those of the transformed, projected set.

Lemma 1**.**

For every finite dataset $D=\{d_{i}:i=1,2,...,n\}\subseteq X$ and an SPSD kernel $K$, there exists an orthonormal Hilbert basis $B$ in the RKHS such that

[TABLE]

where $\delta_{i}\in\mathbb{R}^{n}$ and $\begin{bmatrix}\delta_{1}&\delta_{2}&\cdots&\delta_{n}\end{bmatrix}$ is symmetric and positive semi-definite.

Proof.

We apply Gram-Schmidt orthogonalization process to the maximal linearly independent subset of $\{\Phi_{K}(d_{i}):i=1,2,...,n\}$ and get a set of orthonormal vectors $\{\hat{\beta}_{i}:i=1,2,...,\eta\}$, where

[TABLE]

We may extend this set to a orthonormal Hilbert basis $\hat{B}=\{\hat{\beta}_{i}:i=1,2,...\}$. Then $\forall i\in\{1,2,...,n\}$, $[\Phi_{K}(d_{i})]_{\hat{B}}$ is 0 for all entries after $\eta$ and consequently after $n$, so there exists $\hat{d}_{i}\in\mathbb{R}^{n}$ such that

[TABLE]

As $\begin{bmatrix}\hat{d}_{1}&\hat{d}_{2}&\cdots&\hat{d}_{n}\end{bmatrix}$ is a square matrix, we may compute its singular value decomposition

[TABLE]

We can now define a new orthonormal Hilbert basis $B=\{\beta_{i}:i=1,2,...\}$ through the change of basis matrix $\begin{bmatrix}VU^{T}&0\\ 0&I\end{bmatrix}$. Let $\delta_{i}=VU^{T}\hat{d}_{i}$ for all $i\in\{1,2,...,n\}$, then

[TABLE]

The projected data $\Phi_{K}(d_{i})$ in basis $B$ is $\begin{bmatrix}\delta_{i}^{T}&0\end{bmatrix}^{T}$ and the matrix

[TABLE]

is symmetric and positive definite. ∎

Corollary 1**.**

If we denote $\Psi:V\to\mathbb{R}^{n}$ such that for all $v\in V$,

[TABLE]

or in other words, $\Psi(v)$ is the first $n$ components of $\Phi_{K}(v)$ in the basis $B$. Then for all $d_{i}\in D$,

[TABLE]

that is $\Psi(d_{i})=\delta_{i}$. From the lemma, we have that $\Psi(D)=\begin{bmatrix}\delta_{1}&\delta_{2}&\cdots&\delta_{n}\end{bmatrix}$ is symmetric and positive definite and

[TABLE]

3.1.1 Clustering Equivalences

A clustering setting is a pair $(D,f)$ where $D$ is an finite ordered dataset and $f:D\times D\to\mathbb{R}$ is a measurement on the dataset $D$. We define an equivalence on the clustering setting that $(D_{1},f_{1})=(D_{2},f_{2})$ if and only if $f_{1}(D_{1},D_{1})=f_{2}(D_{2},D_{2})$. For any measurement based clustering method, using measurement $f_{1}$ on $D_{1}$ provides the same exact clustering outcome on the labels as using measurement $f_{2}$ on $D_{2}$. An example of clustering equivalences is that if kernel $K$ corresponds to projection $\Phi_{K}$, then there is $K(D,D)=\langle\Phi_{K}(D),\Phi_{K}(D)\rangle_{\mathbb{H}}$, and therefore $(D,K)=(\phi_{K}(D),\langle\cdot,\cdot\rangle_{\mathbb{H}})$.

3.1.2 Kernel Treelets Equivalences

For a dataset $\{d_{i}:i=1,2,...,n\}$ and a kernel $K$, we already know that there is a clustering equivalence $(D,K)=(\phi_{K}(D),\langle\cdot,\cdot\rangle_{\mathbb{H}})$. From the corollary of lemma 1, there is $\langle\Psi(D),\Psi(D)\rangle=\langle\phi_{K}(D),\phi_{K}(D)\rangle_{\mathbb{H}}$, which provides the equivalence $(\phi_{K}(D),\langle\cdot,\cdot\rangle_{\mathbb{H}})=(\Psi(D),\langle\cdot,\cdot\rangle)$. As $\Psi(D)$ is symmetric, $(\Psi(D),\langle\cdot,\cdot\rangle)=(\Psi^{T}(D),\langle\cdot,\cdot\rangle)$. As a conclusion, $(D,K)=(\Psi^{T}(D),\langle\cdot,\cdot\rangle)$, which implies that a clustering method measured with inner product on dataset $\Psi^{T}(D)$ provides a clustering of $D$ measured with kernel $K$. Therefore, Treelets on $\Psi(D)$ without centering provides a hierarchical clustering of attributes of $\Psi(D)$ based on attribute inner product (covariance matrix), which is a hierarchical clustering of $\Psi^{T}(D)$ based on inner product. According to clustering setting equivalences, this hierarchical clustering is equivalent to a hierarchical clustering of $D$ based on kernel $K$. Furthermore, a property of Treelets is that $\Psi(D)$ does not necessarily need to be computed. The ”covariance matrix” of $\Psi(D)$ without centering has a easier computation method:

[TABLE]

So we may avoid the costly spectral decomposition to compute $\Psi(D)$ and define $A_{0}$ of Treelets as

[TABLE]

3.2 Complexity

The complexity of this algorithm is $O(\xi n_{S}^{2}+n_{S}n_{V})$, where $\xi$ is the complexity of applying kernel function to a pair of data and $\xi=p$ if the data is numeric. In this model, the choice of kernel $K$ determines the expected outcome of the prediction and the choice of sample $S$ determines the variability of the outcome. A small sample size $S$ speeds up the algorithm with the cost of generating false clustering by unrepresentative samples, while large sample size slow down the algorithm and also produces numerical issues because data is more likely to be close to orthogonal as the dimension of projected space grows, and Treelets method would be forced to stop if all remaining components are almost orthogonal. The optimal sample size depends on the floating number accuracy and computation time allowed and should be as large as possible without exceeding the time limit and accuracy limit.

4 Examples

We implemented KT and the following examples in Python with package Numpy [15], Scikit-learn [16], and plots were generated with Matplotlib [17]. The Treelets part of our implementation is not optimized, so it is $O(n^{3})$ runtime in the followings examples rather than $O(n^{2})$ as designed by Lee et al [1]. The hyperparameter $\lambda$ is set to 0 for all the experiments below.

4.1 Clustering for 6 Datasets

To illustrate how KT works as a hierarchical clustering method, we use an example from scikit-learn [16] which consists of 6 datasets, each of which has 1500 two-dimensional data points (i.e. $n=1500$ and $p=2$), and we can visualize each dataset and each cluster by plotting each observation as a point in the plane. Each of the first five datasets consists of data drawn from multiple shapes with an error in distance. The sixth dataset consists of a uniform random sample from $[0,1]^{2}$ to show how clustering method work for uniform distributed data. Figure 1 shows how KT with different kernels works on these datasets compared to the performance of some other clustering methods. The number of clusters and hyper-parameters are tuned for each method and the sample sizes are set to 1000 for each KT method. Each row of this image represents a dataset and each column represents a clustering method. The method each column represents and and its runtime on each dataset is in recorded in Table 1.

In this experiment, KT with RBF kernel is the only method that performs clustering closest to human intuition for all first five datasets. The sixth dataset is a uniform distribution in $[0,1]^{2}$ which we may see how KT is affected by the relative density deficiency in some area due to sampling. Its high performance on the first five datasets is expected as these datasets are to some extent Euclidean distance-based, which corresponds to the assumptions for RBF kernels. Fig.2 shows how difference of number of sample points affects the clustering result. Each column represents KT using RBF kernel with different sample sizes. The hyper-parameter $\sigma=0.1$ is tuned towards $n_{S}=1000$ case and is used for all other sample sizes. Notice that as KT1500 is of full sample size, it does not trigger kernel SVM whereas KT1499 do. Their number of clusters and runtime is recorded in Table 2. From here we can see that more sample data implies more runtime and more stable outcome. The minimum optimal number of samples required for the first 5 datasets are 1000, 100, 1000, 200, 50, respectively, which shows that different datasets requires different amount of samples to explain its shape. Furthermore, the fourth dataset shows that optimal hyper-parameter $\sigma$ is number-of-sample dependent. RBF kernel can be considered as a weighted average of distance and connectivity, where a larger $\sigma$ means a higher weight on distance. For the same $\sigma=0.1$, as sample size gets larger, the clustering result becomes more distance-based rather than connectivity based, demonstrating that optimal $\sigma$ for those sample sizes are actually smaller.

4.2 Clustering for Social Network Dataset

To illustrate how KT works in network analysis we use an example from Stanford Network Analysis Project  [18]. This is a dataset consisting of ’circles’ (or ’friends lists’) from Facebook. It has $n_{V}=4039$ surveyed individual (vertices) and each two of them is connected with vertices if they are friends and not if they are not friends, which are the edges. The edges are undirected and not weighted, and the total number of edges is 88234. We use KT to do clustering on this dataset with full sample size ($S=V$). Denote the set of vertices on the graph as $V$, and define a kernel function $K:V\times V\to\mathbb{R}$ such that

[TABLE]

The number 1045 is computed and chosen as the largest degree of all vertices. Notice that $K$ is a SPSD kernel on $V$ because $K(V,V)$ is a symmetric matrix and is also dominant by the positive diagonal, as $\forall i\in\{1,2,...,n\}$

[TABLE]

To estimate the performance of KT as a multi-scale clustering method on this dataset, we use an evaluation as follows. For each cluster partition in the hierarchy, we compute its matching matrix and its corresponding true positive rate as well as false positive rate. Matching matrix, a type of confusion matrix, is a 2 by 2 matrix recording the number of true positives, true negatives, false positives, and false negatives for pairwise associations. True positive rate measure the proportion of two nodes being in the same cluster given the two nodes are connected and false positive rate measures the proportion of two nodes being in the same cluster given the two nodes are not connected. Each pair of true positive rate and false positive rate produces a point on the plane, and interpolating the set of points of all clustering results in the hierarchy (with order) produces the Receiver operating characteristic (ROC) curve, and the numerical integral over $[0,1]$ interval of this curve is known as Area Under Curve (AUC). Figure 3 demonstrates the performance of KT on the dataset, which provides good clusterings for the dataset because it has an AUC as high as $0.958$.

4.3 Clustering for Dataset with Missing Infomation

To illustrate how KT works on dataset with missing information, we use Mice Protein Expression (MPE) dataset [19] from UCI Machine Learning Repository as an example. This is a dataset consisting of 1080 observations for 8 classes of mice, each of which containing 77 expression levels of different proteins with some of the entries are not avalible. We use KT to do clustering on this dataset. First we normalize these attributes so that each of them has empirical mean 0 and standard deviation 1. Then we define a RBF kernel for dataset with missing data such that for all observation $u,v$,

[TABLE]

Where $E_{uv}$ is the set of indices that is avalible (not missing) in both $u$ and $v$. We check that $E_{uv}\neq\emptyset$ so that it is well-defined. The number 32 is a parameter tuned with experiments. We compare the predicted clusters and the true labels according to pairwise scores. Fig.4 shows how KT performs compared to KMeans clustering. We measure the true positive rate as the proportion of two record being in the same cluster given that they are from mice of the same type, and the false positive rate as the proportion of two record being in the same cluster given that they are from mice of different type. Similar as the example of network dataset, we draw its ROC curve and calculate its AUC. Also, we use KMeans with multiple number of clusters for comparison. The AUC of KT is much higher than the AUC of KMeans ($0.726>0.579$), demonstrating KT is a much better clustering method for this dataset than KMeans.

5 Conclusion

In the paper we describe a novel approach, kernel treelets (KT), for hierarchical clustering. The method relies on applying the treelet algorithm to a matrix measuring similarities among variables in a feature, reproducing kernel Hilbert space. We show with some examples that KT is as useful as other hierarchical clustering methods and is especially competitive for datasets without numerical matrix representation and or missing data. The KT approach also shows significant potential for semi-supervised learning tasks and as a pre-processing, post-processing step in deep-learning. Work in these directions is underway.