Learning Manifolds from Non-stationary Streaming Data

TL;DR

This paper provides theoretical guarantees for manifold learning from streaming data, demonstrating that Gaussian Process Regression can effectively approximate existing methods and detect distribution shifts.

Contribution

It introduces a GPR-based approach with a manifold-specific kernel for streaming manifold learning, including theoretical convergence and change detection capabilities.

Findings

GPR with manifold kernel closely approximates streaming Isomap.

Predictive variance detects shifts in data distribution.

Method effectively learns low-dimensional representations in streaming scenarios.

Abstract

Streaming adaptations of manifold learning based dimensionality reduction methods, such as Isomap, are based on the assumption that a small initial batch of observations is enough for exact learning of the manifold, while remaining streaming data instances can be cheaply mapped to this manifold. However, there are no theoretical results to show that this core assumption is valid. Moreover, such methods typically assume that the underlying data distribution is stationary. Such methods are not equipped to detect, or handle, sudden changes or gradual drifts in the distribution that may occur when the data is streaming. We present theoretical results to show that the quality of a manifold asymptotically converges as the size of data increases. We then show that a Gaussian Process Regression (GPR) model, that uses a manifold-specific kernel function and is trained on an initial batch of sufficient size, can closely approximate the state-of-art streaming Isomap algorithms. The predictive variance obtained from the GPR prediction is then shown to be an effective detector of changes in the underlying data distribution. Results on several synthetic and real data sets show that the resulting algorithm can effectively learn lower dimensional representation of high dimensional data in a streaming setting, while identifying shifts in the generative distribution.