A Simple and Fast Algorithm for L1-norm Kernel PCA

TL;DR

This paper introduces a novel, efficient algorithm for L1-norm kernel PCA that converges quickly, is robust to perturbations, and outperforms benchmarks in outlier detection, with a detailed convergence rate analysis.

Contribution

It proposes a new fixed-point algorithm for L1-norm kernel PCA with proven finite-step convergence and linear rate, addressing non-convexity and non-smoothness issues.

Findings

Algorithm converges to a local optimum in finite steps.

Objective values decrease at a linear rate.

Outperforms benchmarks in outlier detection tasks.

Abstract

We present an algorithm for L1-norm kernel PCA and provide a convergence analysis for it. While an optimal solution of L2-norm kernel PCA can be obtained through matrix decomposition, finding that of L1-norm kernel PCA is not trivial due to its non-convexity and non-smoothness. We provide a novel reformulation through which an equivalent, geometrically interpretable problem is obtained. Based on the geometric interpretation of the reformulated problem, we present a fixed-point type algorithm that iteratively computes a binary weight for each observation. As the algorithm requires only inner products of data vectors, it is computationally efficient and the kernel trick is applicable. In the convergence analysis, we show that the algorithm converges to a local optimal solution in a finite number of steps. Moreover, we provide a rate of convergence analysis, which has been never done for any L1-norm PCA algorithm, proving that the sequence of objective values converges at a linear rate. In numerical experiments, we show that the algorithm is robust in the presence of entry-wise perturbations and computationally scalable, especially in a large-scale setting. Lastly, we introduce an application to outlier detection where the model based on the proposed algorithm outperforms the benchmark algorithms.