The Geometry of the Pivot: A Note on Lazy Pivoted Cholesky and Farthest Point Sampling

TL;DR

This paper clarifies the geometric interpretation of the Pivoted Cholesky algorithm in kernel methods, showing its equivalence to Farthest Point Sampling and Gram-Schmidt orthogonalization within the RKHS, with practical implementation insights.

Contribution

It provides a geometric perspective on Pivoted Cholesky, linking it to Farthest Point Sampling and Gram-Schmidt, and offers a simple Python implementation.

Findings

Pivoted Cholesky is equivalent to Farthest Point Sampling in RKHS.

Cholesky factorization corresponds to Gram-Schmidt orthogonalization.

Provides a clear derivation and practical Python code.

Abstract

Low-rank approximations of large kernel matrices are ubiquitous in machine learning, particularly for scaling Gaussian Processes to massive datasets. The Pivoted Cholesky decomposition is a standard tool for this task, offering a computationally efficient, greedy low-rank approximation. While its algebraic properties are well-documented in numerical linear algebra, its geometric intuition within the context of kernel methods often remains obscure. In this note, we elucidate the geometric interpretation of the algorithm within the Reproducing Kernel Hilbert Space (RKHS). We demonstrate that the pivotal selection step is mathematically equivalent to Farthest Point Sampling (FPS) using the kernel metric, and that the Cholesky factor construction is an implicit Gram-Schmidt orthogonalization. We provide a concise derivation and a minimalist Python implementation to bridge the gap between theory and practice.