Compressed Diffusion

TL;DR

This paper introduces a novel compressed diffusion method that reduces computational complexity in diffusion maps by using data regions instead of individual data points, enabling efficient analysis of large datasets.

Contribution

It presents a new approach to compute diffusion geometry via data region compression, improving scalability and efficiency over traditional diffusion maps.

Findings

Outperforms existing methods on large datasets

Reduces computational complexity from cubic to lower orders

Provides accurate diffusion embeddings through data region approximation

Abstract

Diffusion maps are a commonly used kernel-based method for manifold learning, which can reveal intrinsic structures in data and embed them in low dimensions. However, as with most kernel methods, its implementation requires a heavy computational load, reaching up to cubic complexity in the number of data points. This limits its usability in modern data analysis. Here, we present a new approach to computing the diffusion geometry, and related embeddings, from a compressed diffusion process between data regions rather than data points. Our construction is based on an adaptation of the previously proposed measure-based Gaussian correlation (MGC) kernel that robustly captures the local geometry around data points. We use this MGC kernel to efficiently compress diffusion relations from pointwise to data region resolution. Finally, a spectral embedding of the data regions provides coordinates that are used to interpolate and approximate the pointwise diffusion map embedding of data. We analyze theoretical connections between our construction and the original diffusion geometry of diffusion maps, and demonstrate the utility of our method in analyzing big datasets, where it outperforms competing approaches.

Equations

$$k_{\mathcal{S}}(S,T)={\int }_S{\int }_{T}k(x,y)dxdy,S,T\in \mathcal{S},$$

$$[K_{\mathcal{S}}{]}_{ij}=k(S_i,S_j),i,j=1,\dots ,n.$$

$$[D^t(x,y){]}^2=D_2^t(x,y)$$

$$=\sum _{j=0}^{\infty }{\lambda }_j^{2t}{\left[{\psi }_j(x)-{\psi }_j(y)\right]}^2$$

$${\Vert {\mu }_{\ell }(x)-{\rho }_t(x)\Vert }_2^2\propto \left|\ell -\sum _{j=0}^{|X|-1}{\lambda }_j^{2t}\right|$$

$$D_c^t(x,y)=\arccos \left(\frac{{\sum }_{j=0}^{\infty }{\lambda }_j^{2t}{\psi }_j(x){\psi }_j(y)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}\right).$$

$$S_i=\{x\in X:D_c^t(x,y_i)=\underset{y\in L}{\mathrm{argmin}}{D}_c^t(x,y)\}.$$

$$P_{\mathcal{S}}(S,T)$$

$$={\iint }_S{\int }_T\frac{g(x,r)}{\Vert k_{\mathcal{S}}(S,\cdot ){\Vert }_1}\frac{g(r,y)}{\Vert g(r,\cdot ){\Vert }_1}dydxdr={\iint }_S\frac{g(x,r)}{\Vert k_{\mathcal{S}}(S,\cdot ){\Vert }_1}dx{\int }_T\frac{g(r,y)}{\Vert g(r,\cdot ){\Vert }_1}dydr$$

$$={\iint }_{S}p(x,r)\frac{\Vert g(x,\cdot ){\Vert }_1}{\Vert k_{\mathcal{S}}(S,\cdot ){\Vert }_1}dx{\int }_{T}p(r,y)dydr={\iint }_{S}p(x,r)\mathrm{Pr}[x|S]dx\mathrm{Pr}[r\stackrel{1\text{ step}}{\text{longleadsto}}T]dr$$

$$=\int \mathrm{Pr}[S\stackrel{1\text{ step}}{\text{longleadsto}}r]\mathrm{Pr}[r\stackrel{1\text{ step}}{\text{longleadsto}}T]dr\text{qed}$$

$$[I_{\ell }{]}_{ii}=\{\begin{array}{ll}1& i\le \ell \\ 0& i>\ell .\end{array}$$

$$\left|\sum _{i=1}^N{\mu }_{\ell }(x_i)-{\rho }_t(x_i)\right|$$

$$=\left|\text{trace}\left(\Psi I_{\ell }^N{\Psi }^T-A^{2t}\right)\right|=\left|\text{trace}\left[\Psi \left(I_{\ell }^N-{\Lambda }^{2t}\right){\Psi }^T\right]\right|$$

$$=\left|\text{trace}\left(I_{\ell }^N-{\Lambda }^{2t}\right)\right|=\left|\ell -\sum _{j=0}^{|X|-1}{\lambda }_j^{2t}\right|$$

$$\frac{{\sum }_{j=0}^{\infty }{\lambda }_j^{2t}{\psi }_j(x_i){\psi }_j(x_j)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}=⟨\frac{a^t(x,\cdot )}{\Vert a^t(x,\cdot ){\Vert }_2},\frac{a^t(y,\cdot )}{\Vert a^t(y,\cdot ){\Vert }_2}⟩\in [0,1],x,y\in X,$$

$$\underset{y\in \mathcal{M}}{\mathrm{argmin}}{D}_c^t(x,y)=\underset{y\in \mathcal{M}}{\mathrm{argmax}}\frac{{\sum }_{j=0}^{\infty }{\lambda }_j^{2t}{\psi }_j(x_i){\psi }_j(x_j)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}=\underset{y\in \mathcal{M}}{\mathrm{argmax}}\frac{a^{2t}(x,y)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}.$$

$${\Vert \frac{a^t(x,\cdot )}{\sqrt{{\rho }_t(x)}}-\frac{a^t(y,\cdot )}{\sqrt{{\rho }_t(y)}}\Vert }_2^2=\frac{\Vert a^t(x,\cdot ){\Vert }_2^2}{{\rho }_t(x)}+\frac{\Vert a^t(y,\cdot ){\Vert }_2^2}{{\rho }_t(y)}-2⟨\frac{a^t(x,\cdot )}{\sqrt{{\rho }_t(x)}},\frac{a^t(y,\cdot )}{\sqrt{{\rho }_t(y)}}⟩=2-2\frac{a^{2t}(x,y)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}.$$

$$\underset{y\in \mathcal{M}}{\mathrm{argmin}}{\Vert \frac{a^t(x,\cdot )}{\sqrt{{\rho }_t(x)}}-\frac{a^t(y,\cdot )}{\sqrt{{\rho }_t(y)}}\Vert }_2^2=\underset{y\in \mathcal{M}}{\mathrm{argmin}}2-2\frac{a^{2t}(x,y)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}=\underset{y\in \mathcal{M}}{\mathrm{argmax}}\frac{a^{2t}(x,y)}{\sqrt{{\rho }_t(x){\rho }_t(y)}}=\underset{y\in \mathcal{M}}{\mathrm{argmin}}{D}_c^t(x,y),$$

Full text

I Introduction

Manifold learning approaches are often used for modeling and uncovering intrinsic low dimensional structure in high dimensional data (e.g., in genomics [6]). Diffusion maps [3], in particular, are a popular method that capture data manifolds with random walks that propagate through nonlinear pathways in the data. Transition probabilities of a Markovian diffusion process define an intrinsic diffusion distance metric that is amenable to low dimensional embedding. Indeed, by arranging transition probabilities in a row-stochastic diffusion operator, and taking its leading eigenvalues and eigenvectors, one can derive a small set of coordinates where diffusion distances are approximated as Euclidean distances, and intrinsic manifold structures are revealed.

While the embedding provided by diffusion maps is useful for data analysis, it is also challenging to apply to modern “big data” settings due to scalability issues. As is often the case with kernel methods, diffusion maps require the computation of pointwise transition probabilities, which is computationally quadratic in the size of the data, and an eigendecomposition which is cubic if all eigenvectors are computed. Here, we show that one can reduce the computational cost significantly by only considering transition probabilities between data regions, which can be efficiently computed from a compressed affinity kernel over a fixed-size partition of the data. Our construction is based on coarse-graining a measure-based kernel [1, 2] with an inverse-density measure, which was recently introduced in [4] for geometry-based data generation and uniform resampling of data manifolds. Here we extend the uses of this kernel to provide efficient implementation and application of diffusion maps by first embedding data regions, and then interpolating the pointwise embedding from their diffusion coordinates.

The main contributions of this work are as follows. On the theoretical side, we further establish the relations between the original diffusion framework from [3] and the construction in [4], both at a pointwise and compressed (i.e., data-region) level. On the practical side, we suggest a novel partitioning method, the results of which indicate significant speedups in the computation of the diffusion embedding, which outperform other approaches, and enables the application of diffusion-based manifold learning well beyond the data sizes traditionally used with kernel methods.

II Problem setup

II-A Preliminaries

Let $\mathcal{M}\subseteq\mathbbm{R}^{m}$ be a compact $d$ dimensional manifold immersed in the ambient space $\mathbbm{R}^{m}$, where $d\ll m$, which represents the intrinsic geometry of data sampled from it. For simplicity, the integration notation $\int\cdot\,dy$ in this paper will refer to the Lebesgue integral $\int_{\mathcal{M}}\cdot\,dy$ over the manifold, instead of the whole space $\mathbbm{R}^{n}$. Further, while (for simplicity) such integrals are written without a specific measure one can equivalently, w.l.o.g., replace $dx$ with an appropriate measure representing data sampling distribution over $\mathcal{M}$. Let $g(x,y)\triangleq\exp\left(-\|x-y\|^{2}/\varepsilon\right)$, $x,y\in\mathcal{M}$, $\varepsilon>0$, define the Gaussian kernel $Gf(x)=\int g(x,y)f(y)dy$ used in [3] to capture local neighborhoods from data sampled from $\mathcal{M}$. Following [3] and related work, we define the Gaussian degree $q(x)=\left\|g(x,\cdot)\right\|_{1}=\int g(x,y)dy$ and assume it provides a suitable approximation of the distribution (or local density) of data over the manifold $\mathcal{M}$. Finally, given a measure $\mu$ over the manifold, an MGC kernel [1, 2] is defined as $k_{\mu}(x,y)=\int g(x,r)g(y,r)d\mu(r)$. Note that while we use a Gaussian kernel for the remainder of this work, the definitions and theorems to follow do not depend on the choice of $g$, so long as it is a kernel function.

The original MGC construction in [2] considered measures that represent data distribution, and used the constructed MGC kernel to define diffusion maps (see Sec. II-B and [3]) with them. Recently, it was shown in [4] that an MGC kernel constructed with inverse-density measure allows separation of data geometry and distribution, which in turn allows uniform data generation over the data manifold, with applications in alleviating sampling biases in data analysis (e.g., imbalanced classification). Here, we further explore the inverse-density MGC kernel and its properties. In particular, we show this kernel enables compression of its resulting diffusion geometry into data regions, instead of data points, to efficiently capture the intrinsic manifold geometry of analyzed data.

II-B Diffusion Maps

Diffusion maps [3] utilize a set of local affinities to define a Markovian diffusion process over analyzed data, which captures the intrinsic data geometry via diffusion distances. The original construction [3] defines transition probabilities between data points based on Gaussian affinities as $p(x.y)=g(x,y)/q(x)$, where it can be verified that $\int p(x,y)dy=1$. This construction can also be generalized to other affinity kernels, such as the MGC kernel $k_{\mu}(x,y)$ from [2] or variation in [4]. Under mild conditions on the affinity kernel, the resulting transition probability operator has a discrete decaying spectrum of eigenvalues $1=\lambda_{0}\geq\left|\lambda_{1}\right|\geq\left|\lambda_{2}\right|\geq\ldots$ which are used together with their corresponding eigenfunctions $\phi_{0},\phi_{1},\phi_{2},\ldots$ (with $\phi_{0}$ being constant) to achieve the diffusion map of the data. Each data point $x\in\mathcal{M}$ is embedded by this diffusion map to the diffusion coordinates $\Phi_{t}(x)=(\lambda_{1}^{t}\phi_{1}(x),\ldots,\lambda^{t}_{\delta}(x)\phi_{\delta}(x))$, where the exact value of $\delta$ depends on the spectrum of the transition probabilities operator $Pf(x)=\int p(x,y)f(y)dy$, whose kernel is $p(x,y)$.

III Inverse-density MGC kernel

Our construction here is based on an inverse-density MGC kernel, first introduced in [4], which is defined as follows:

Definition 1**.**

The inverse-density MGC (ID-MGC) kernel is defined as $k(x,y)=\int\frac{g(x,r)g(r,y)}{q(r)}dr$, $x,y\in\mathcal{M}$, with the associated integral operator $Kf(x)=\int k(x,y)f(y)dy$.

This kernel corresponds to $k_{\mu}(x,y)$ with $d\mu(x)=q^{-1}(x)dx$, where in practice $q(x)$ accounts (up to normalization) for data density over $\mathcal{M}$. Therefore, the connectivity captured by this kernel normalizes density variations by enhancing relations in sparse regions compared to dense ones. Indeed, in [4] this property was used to adjust the distribution of data and provide uniform resampling of data manifolds. We note that since we consider $\mathcal{M}$ as representing the intrinsic geometry of collected (or observed) data, we expect a certain amount of data points to exist in each local region, and thus $q^{-1}(x)$ can be expected to have a finite upper bound over $\mathcal{M}$.

Several useful properties of the ID-MGC kernel and its relation to the Gaussian-based diffusion operator in Sec. II-B are summarized in the following theorem (proof of this proposition is found in Appendix A):

Theorem 1**.**

*Let the kernel $k(\cdot,\cdot)$ and operator $K$ be defined as in Def. 1. Then, this kernel (and operator) can be related to the Gaussian kernel $G$ and diffusion operator $P$ via

1. the operator norm: $\|K\|\leq\|G\|$,
2. the kernel degrees: $\|k(x,\cdot)\|_{1}=\|g(x,\cdot)\|_{1}$, $x\in\mathcal{M}$, and
3. two-step diffusion: $P^{2}f(x)=\int\frac{k(x,y)}{\|k(x,\cdot)\|_{1}}f(y)dy$.

IV Compressed kernel and diffusion transitions

Let $S_{1},\ldots,S_{n}\subset\mathcal{M}$ be a partition of the manifold into measurable subsets, such that $\mathcal{M}=\bigcup_{j=1}^{n}S_{j}$, and $S_{i}\cap S_{j}=\emptyset$ for every $i\neq j$. We define a compressed kernel over such a partition as follows:

Definition 2**.**

Let $k(x,y)$, $x,y\in\mathcal{M}$ be defined as in Def. 1. The compressed kernel over partition $\mathcal{S}=\{S_{j}\}_{j=1}^{n}$ of $\mathcal{M}$ is given by

[TABLE]

with the corresponding $n\times n$ kernel matrix given by

[TABLE]

Similar to the original diffusion map construction in Sec. II-B, we normalize the compressed kernel to get diffusion probabilities $p_{\mathcal{S}}(S,T)=k(S,T)/\|k(S,\cdot)\|_{1}$, organized in an $n\times n$ row-stochastic matrix $P_{\mathcal{S}}$. This matrix captures diffusion transition probabilities between data regions. The relation between the compressed construction and the original diffusion framework is summarized in the following theorem (proof of this proposition is found in Appendix B):

Theorem 2**.**

*The compressed construction here can be related to the pointwise one in Sec. II-B, via

1. operator norm (in matrix & operator form): $\|K_{\mathcal{S}}\|\leq O(\|G\|)$ with a constant that only depends on the finite volume of $\mathcal{M}$
2. kernel degrees: $\|k_{\mathcal{S}}(S,\cdot)\|_{1}=\int_{S}\|g(x,\cdot)\|_{1}dx$, $S\in\mathcal{S}$, and
3. diffusion probabilities: $P_{\mathcal{S}}(S,T)=\int\Pr[S\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}r]\Pr[r\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}T]\,dr$ where $\Pr[r\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}T]=\int_{T}p(r,y)dy$ and $\Pr[S\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}r]=$ $\int_{S}p(x,r)\Pr[x|S]dx$ with prior $\Pr[x|S]=\frac{\|g(x,\cdot)\|_{1}}{\|k_{\mathcal{S}}(S,\cdot)\|_{1}}$.

V Compression-based fast diffusion maps

The compressed construction of $K_{\mathcal{S}}$ and $P_{\mathcal{S}}$ gives rise to a natural approximation of diffusion maps. Given a partitioning $\mathcal{S}$, we define a compressed diffusion map $\Phi^{t}_{\mathcal{S}}:\mathcal{S}\to\mathbbm{R}^{\delta}$ analogous to the pointwise one in II-B using eigenvectors of $P_{\mathcal{S}}$ (with corresponding eigenvalues) in lieu of the eigenfunctions of $P$. This diffusion map provides an embedding of data regions, rather than points, which represents a coarse version of the diffusion geometry over $\mathcal{M}$. Then, using the region-to-point transition probabilities $Pr[S\stackrel{{\scriptstyle 1step}}{{\longleadsto}}x]$ from Theorem 2, we approximate the pointwise diffusion map as $\widetilde{\Phi}^{t}(x)=\sum_{j=1}^{n}\Phi^{t}_{\mathcal{S}}(S_{j})Pr[S_{j}\stackrel{{\scriptstyle 1step}}{{\longleadsto}}x]$. Results in Sec. VI indicate that with the proper choice of partitions in $\mathcal{S}$, the compressed diffusion map $\widetilde{\Phi}^{t}$ provides a good approximation of the original pointwise one from [3], while also providing significant scalability and performance advantages over both direct computation and other alternative approximations.

It now remains to derive a strategy for efficiently partitioning the data manifold such that compression over $\mathcal{S}$ will suitably capture (albeit at a coarser level) the pointwise diffusion geometry defined by $P$. To this end, it is convenient to consider diffusion affinities and distances [3, 1, 2], rather than transition probabilities. These are defined as follows:

Definition 3**.**

The diffusion affinity [3] kernel is the symmetric conjugate of $P$, which is defined spectrally as $Af(x)=\sum_{j=1}^{\infty}\lambda_{j}\langle\psi_{j},f\rangle\psi_{j}(x)$ with $\psi_{j}=q^{1/2}(x)\phi_{j}(x)$.

Definition 4**.**

The diffusion distance [3] between two points $x$ and $y$ sampled from $\mathcal{M}$ is defined via the kernel function $a^{t}(\cdot,\cdot)$ of $A^{t}$, and equivalently via its eigendecomposition, as

[TABLE]

We note that in the described compressed diffusion scheme, we replace pointwise affinities with affinities between data points and partitions, rather than just between partitions. Under proper partitioning of the data, we expect (or assume) such substitution would retain intrinsic structure of the data, as captured by the diffusion geometry defined by $P$ or $A$. The following proposition examines the difference between pointwise and coarse-grained diffusion affinity information, and relates this difference to locality of partitions with respect to the diffusion distance metric.

Proposition 3**.**

Let $S\in\mathcal{S}$ and $\xi>0$. If the diffusion affinities satisfy $\sup\limits_{z\in\mathcal{M},x\in S_{j}}\left|a(x,z)-\mathbb{E}_{y\in S_{j}}[a(y,z)]\right|<\xi$ , then the diffusion distances $D^{1}(x,y)$, $x,y\in S$, are bounded from above by $O(\xi)$, with a constant that only depends on $\operatorname{vol}{\mathcal{M}}$.

Proof.

For any $x,y\in S_{j}$ and $z\in\mathcal{M}$, due to the triangle inequality, we have $|a(x,z)-a(y,z)|\leq|a(x,z)-\mathbb{E}_{u\in S_{j}}[a(u,z)]|+|a(y,z)-\mathbb{E}_{u\in S_{j}}[a(u,z)]|\leq 2\varepsilon$. Therefore, $\int|a(x,z)-a(y,z)|^{2}dz\leq 4\varepsilon^{2}\operatorname{vol}{\mathcal{M}}$, which together with Definition 4 yields the result in the proposition. ∎

Indeed, this result indicates that to faithfully capture the diffusion geometry by given partitioning $\mathcal{S}$, the partitions should be local in the diffusion geometry, since the diffusion geometry in turn preserves the local geometry of the manifold [3]. This understanding motivates our formulation of a partitioning strategy, as explained below.

Here we turn to practical settings of data analysis applications. Therefore, in the following we consider finite settings, particularly for some dataset $X\subset\mathcal{M}$, $|X|<\infty$, sampled from the manifold, with $P$ and $A$ being $|X|\times|X|$ matrices constructed from it. Note also that in the this setting, the eigenfunctions of $P$ or $A$ become eigenvectors (i.e., in $\mathbbm{R}^{|X|}$), and the RHS in Definition 4 is written analogously with the sum going over $|X|$ eigenpairs.

In order to define a partition on $X$, we first choose a set of partition centroids $L$. Noting the success of coherence-based sampling strategies for signal compression and recovery[7], we propose a random sampling without replacement biased by coherence, where we estimate the coherence as follows.

Proposition 4**.**

Let $\mu_{\ell}(x)=\sum_{j=0}^{\ell}|\psi_{j}(x)|^{2}$ be the order-$\ell$ coherence of $x\in X$ as in [7]. Then, the t-step diffusion coherence $\rho_{t}(x)=\|a^{t}(x,\cdot)\|_{2}^{2}$ is an estimate of $\mu_{\ell}(x)$ with error

[TABLE]

for every $x\in X$.

Proof of this proposition can be seen in Appendix C.

According to Proposition 3, we should assign $x\in X$ to the partition associated with centroid $y\in L$ such that $D^{t}_{2}(x,y)$ is minimized. However, this distance is in practice biased by the diffusion coherence $\rho_{t}(y)$. We therefore define the angular diffusion distance, which we will show maximizes transition probabilities between points and the assigned centroids.

Definition 5**.**

The angular diffusion distance between $x,y\in\mathcal{M}$ is defined using the eigendecomposition of the diffusion affinity kernel from Def. 3 as

[TABLE]

Proposition 5**.**

For $x\in\mathcal{M}$, the following dualities hold:

$\displaystyle\operatorname*{\arg\,\min}_{y\in\mathcal{M}}~{}D^{t}_{c}(x,y)=\operatorname*{\arg\,\max}_{y\in\mathcal{M}}~{}\frac{a^{2t}(x,y)}{\sqrt{\rho_{t}(x)\,\rho_{t}(y)}}$* ;* 2. 2.

$\displaystyle\operatorname*{\arg\,\min}_{y\in\mathcal{M}}~{}\left\|\frac{a^{t}(x,\cdot)}{\sqrt{\rho_{t}(x)}}-\frac{a^{t}(y,\cdot)}{\sqrt{\rho_{t}(y)}}\right\|_{2}^{2}=\operatorname*{\arg\,\min}_{y\in\mathcal{M}}~{}D_{c}^{t}(x,y)$* .*

Proof of this proposition can be seen in Appendix D. Propositions 4 and 5 provide a convenient method to select a set of partitions that optimize proposition 3. To do this, we assign points $x\in X$ to partitions $S_{i}\in\mathcal{S}$ corresponding to centroids $y_{i}\in L$ according to

[TABLE]

VI Empirical Results

VI-A Swiss roll

Here we compare three techniques for approximation of diffusion maps: linear interpolated diffusion on cluster centroids, volume-weighted Nystrom extension[5], and our proposed compression-based fast diffusion map (CFDM).

The Swiss roll is a canonical test dataset for diffusion maps, and manifold learning in general. Figure V.1 shows the diffusion map of the Swiss roll and each approximation, with the partition diffusion coordinates plotted in green. To quantify the reconstruction, we calculated the sum of squared error (SSE) across 32 diffusion components of 200 Swiss rolls, allowing for sign flips and reordering of components. CFDM produces a lower approximation error and a smaller approximate kernel for less computational cost than competing methods (Fig. VI.1). For $n\geq 10^{4}$, CFDM offers an approximately 10x speed-up over exact diffusion maps, and this advantage grows slightly as $n$ increases.

VI-B Mass cytometry of induced pluripotent stem cells

Mass cytometry is the measurement of protein abundance in individual cells via mass spectrometry. We used CFDM to approximate the diffusion map of 20,000 single cells in an induced pluripotent stem cell (iPSC) system[8] with 500 partitions. Figure VI.2 shows the exact and approximated diffusion maps. CFDM produces a visually equivalent embedding in a fraction of the time, accurately revealing the differentiation of skin cells into both iPSCs and a failed reprogramming state. Such an approximation will be beneficial for constructing diffusion maps on large graphs, such as social networks, as well as for downstream applications, where algorithms of high computational complexity are run on the diffusion map.

Appendix A Proof of Theorem 1

We prove the theorem in three parts, via the following lemmas. We start with the kernel degrees:

Lemma 6**.**

$\|k(x,\cdot)\|_{1}=\|g(x,\cdot)\|_{1}$**

Proof.

Direct computation yields $\displaystyle\|k(x,\cdot)\|_{1}=\int k(x,y)dy=\iint\frac{g(x,r)}{\|g(x,\cdot)\|_{1}}g(r,y)drdy=\int\frac{g(x,r)}{\|g(x,\cdot)\|_{1}}\underbrace{\int g(r,y)dy}_{\|g(x,\cdot)\|_{1}}dr=\int g(x,r)dr=\|g(x,\cdot)\|_{1}.$ ∎

Next, we prove the relation to diffusion transition probabilities:

Lemma 7**.**

$P^{2}f(x)=\frac{1}{\|k(x,\cdot)\|_{1}}\int k(x,y)f(y)dy$**

Proof.

Direct computation yields $\displaystyle P^{2}f(x)=\int p(x,r)\int p(r,y)f(y)dydr=\frac{1}{\|g(x,\cdot)\|_{1}}\int\frac{g(x,r)}{\|g(r,\cdot)\|_{1}}\int g(r,y)f(y)dydr=\frac{1}{\|g(x,\cdot)\|_{1}}\int\underbrace{\int\frac{g(x,r)}{\|g(r,\cdot)\|_{1}}g(r,y)dr}_{k(x,y)}f(y)dy=\frac{1}{\|k(x,\cdot)\|_{1}}\int k(x,y)f(y)dy.$ ∎

Finally, we consider the operator norm:

Lemma 8**.**

$\|K\|\leq\|G\|$**

Proof.

Let $Qf(x)=q(x)f(x)$; thus, we can verify $G=QP$ and by combining the previous two lemmas we get $\|K\|=\|QP^{2}\|=\|GP\|\leq\|G\|\|P\|=\|G\|$, since $P$ is row-stochastic with $\|P\|=1$. ∎

Appendix B Proof of Theorem 2

We prove the theorem in three parts, via the following lemmas. We start with the operator norm:

Lemma 9**.**

$\|K_{\mathcal{S}}\|\leq[\operatorname{vol}{\mathcal{M}}]^{3/2}\|G\|$**

Proof.

Let $v\in\mathbbm{R}^{n}$, and define the simple function $f_{v}=\sum_{j=1}^{n}[v]_{j}\chi_{S_{j}}$, where $\chi_{S_{j}}$ is the characteristic (i.e., indicator) function of $S_{j}$. Notice that since $\mathcal{M}$ is compact with finite volume, then $\|f\|_{2}=\left[\sum_{j=1}^{n}[v]_{j}^{2}\operatorname{vol}{S_{j}}\right]^{1/2}\leq[\operatorname{vol}{\mathcal{M}}]^{1/2}\|v\|_{2}$. Further, for $i=1,\ldots,n$, $\displaystyle\left[K_{\mathcal{S}}v\right]_{i}=\sum_{j=1}^{n}\left[\int_{S_{i}}\int_{S_{j}}k(x,y)dydx\right][v]_{j}=\int_{S_{i}}\int k(x,y)f_{v}(y)dydx=\int_{S_{i}}Kf(x)dx$. Therefore, $\displaystyle\|K_{\mathcal{S}}v\|_{2}^{2}=\sum_{i=1}^{n}\left|\int_{S_{i}}Kf(x)dx\right|^{2}\leq\left|\int Kf(x)dx\right|^{2}=\|Kf\|_{1}^{2}\leq[\operatorname{vol}{\mathcal{M}}]^{2}\|Kf\|_{2}^{2}$ where the last inequality is due to Hölder inequality. Finally, by combining these results with Theorem 1 we get $\displaystyle\|K_{\mathcal{S}}v\|_{2}\leq\operatorname{vol}{\mathcal{M}}\|K\|\|f\|_{2}\leq[\operatorname{vol}{\mathcal{M}}]^{3/2}\|G\|\|v\|_{2}$ for every $v\in\mathbbm{R}^{n}$, which yields the result in the lemma. ∎

Next, we consider the diffusion degrees:

Lemma 10**.**

$\|k_{\mathcal{S}}(S,\cdot)\|_{1}=\int_{S}\|g(x,\cdot)\|_{1}dx$* for all $S\in\mathcal{S}$*

Proof.

Direct computation yields $\displaystyle\|k_{\mathcal{S}}(S,\cdot)\|_{1}=\sum_{j}k_{\mathcal{S}}(S,S_{j})=\int_{S}\left[\sum_{j}\int_{S_{j}}k(x,y)dy\right]dx=\int_{S}\int k(x,y)dy\,dx=\int_{S}\|k(x,\cdot)\|_{1}dx$, and together with Theorem 1 we get the result in the lemma. ∎

Finally, we prove the relation to diffusion transition probabilities:

Lemma 11**.**

$P_{\mathcal{S}}(S,T)=\int\Pr[S\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}r]\Pr[r\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}T]\,dr$* where $\Pr[r\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}T]=\int_{T}p(r,y)dy$ and $\Pr[S\stackrel{{\scriptstyle 1\text{ step}}}{{\longleadsto}}r]=\int_{S}p(x,r)\Pr[x|S]dx$ with prior $\Pr[x|S]=\frac{\|g(x,\cdot)\|_{1}}{\|k_{\mathcal{S}}(S,\cdot)\|_{1}}$*

Proof.

Direct computation yields:

[TABLE]

Appendix C Proof of Proposition 4

Let $\Psi$ be a matrix whose columns are the eigenvectors $\psi_{j}$, $j=0,\ldots,|X|-1$, of the $A$ (in finite settings), and $\Lambda$ be a diagonal matrix that consists of the corresponding eigenvalues $1=\lambda_{0}\geq\cdots\lambda_{|X|-1}$. Since $A$ is symmetric, it yields an orthonormal set of eigenvectors, thus $\Psi$ is an orthogonal matrix and further, we can write $A^{t}=\Psi\Lambda^{t}\Psi^{T}$ for any $t\geq 0$. Let $I_{\ell}$ be an $N\times N$ diagonal matrix in which

[TABLE]

Finally, since we are considering finite settings, we can enumerate that dataset as $X=\{x_{1},\ldots,x_{N}\}$, and thus the order-$\ell$ coherence defined in Prop. 4 as $\mu_{\ell}(x_{i})=[\Psi I_{\ell}\Psi^{T}]_{ii}$ for $i=1,\ldots,N$. Similarly, we can also write $\rho_{t}(x_{i})=\|a^{t}(x_{i},\cdot)\|_{2}^{2}=\langle a^{t}(x_{i},\cdot),a^{t}(x_{i},\cdot)\rangle=[A^{2t}]_{ii}$, $i=1,\ldots,N$.

With this matrix formulation, we now show the stated result $\left|\mu_{\ell}(x)-\rho_{t}(x)\right|\propto\left|\ell-\sum_{j=0}^{|X|-1}\lambda_{j}^{2t}\right|$ from the proposition by direction computation, as:

[TABLE]

where (C.1) is due to the eigendecomposition of $A^{2t}$ and (C.2) is due to the orthogonality of $\Psi$. ∎

Appendix D Proof of Proposition 5

We note that as shown in [3], the spectral theorem yields $\sum_{j=0}^{\infty}\lambda_{j}^{2t}\psi_{j}(x_{i})\psi_{j}(x_{j})=a^{2t}(x,y)=\langle a^{t}(x,\cdot),a^{t}(y,\cdot)\rangle$ for every $t\geq 0$. Furthermore, since by definition $\rho_{t}(x)=\|a^{t}(x,\cdot)\|_{2}^{2}$, we have

[TABLE]

as an inner product of normalized (i.e., unit-norm ) non-negative functions. Therefore, since $\arccos$ is monotonically decreasing in the interval $[0,1]$, we get the first duality. Namely, for each $x\in\mathcal{M}$, $t\geq 0$,

[TABLE]

To show the second duality, we first write

[TABLE]

Therefore, together with the first duality, we now get

[TABLE]

which completes the proof. ∎