Revisiting Memory Efficient Kernel Approximation: An Indefinite Learning Perspective

TL;DR

This paper extends the MEKA kernel approximation method to non-stationary and indefinite kernels, providing theoretical insights and practical algorithms for stable, positive semi-definite approximations in large-scale machine learning.

Contribution

It introduces a generalized MEKA framework applicable to a wider class of kernels, including indefinite ones, with a Lanczos-based spectrum shift for stability and positive semi-definiteness.

Findings

Extended MEKA to non-stationary kernels like polynomial and extreme learning kernels.

Developed a Lanczos-based spectrum shift for stable positive semi-definite approximations.

Validated the approach with experiments on synthetic and real-world datasets.

Abstract

Matrix approximations are a key element in large-scale algebraic machine learning approaches. The recently proposed method MEKA (Si et al., 2014) effectively employs two common assumptions in Hilbert spaces: the low-rank property of an inner product matrix obtained from a shift-invariant kernel function and a data compactness hypothesis by means of an inherent block-cluster structure. In this work, we extend MEKA to be applicable not only for shift-invariant kernels but also for non-stationary kernels like polynomial kernels and an extreme learning kernel. We also address in detail how to handle non-positive semi-definite kernel functions within MEKA, either caused by the approximation itself or by the intentional use of general kernel functions. We present a Lanczos-based estimation of a spectrum shift to develop a stable positive semi-definite MEKA approximation, also usable in classical convex optimization frameworks. Furthermore, we support our findings with theoretical considerations and a variety of experiments on synthetic and real-world data.