Pack only the essentials: Adaptive dictionary learning for kernel ridge regression

TL;DR

This paper introduces SQUEAK, an improved algorithm for kernel ridge regression that reduces space complexity by efficiently approximating ridge leverage scores without the need for normalization or estimating effective dimension.

Contribution

SQUEAK simplifies and enhances INK-Estimate by using unnormalized RLS, achieving near-optimal space complexity for large-scale kernel ridge regression.

Findings

SQUEAK achieves space complexity close to exact RLS sampling.

It simplifies the process by removing the need for effective dimension estimation.

The algorithm maintains accuracy with fewer columns in Nystrom approximations.

Abstract

One of the major limits of kernel ridge regression (KRR) is that storing and manipulating the kernel matrix K_n for n samples requires O(n^2) space, which rapidly becomes unfeasible for large n. Nystrom approximations reduce the space complexity to O(nm) by sampling m columns from K_n. Uniform sampling preserves KRR accuracy (up to epsilon) only when m is proportional to the maximum degree of freedom of K_n, which may require O(n) columns for datasets with high coherence. Sampling columns according to their ridge leverage scores (RLS) gives accurate Nystrom approximations with m proportional to the effective dimension, but computing exact RLS also requires O(n^2) space. (Calandriello et al. 2016) propose INK-Estimate, an algorithm that processes the dataset incrementally and updates RLS, effective dimension, and Nystrom approximations on-the-fly. Its space complexity scales with the effective dimension but introduces a dependency on the largest eigenvalue of K_n, which in the worst case is O(n). In this paper we introduce SQUEAK, a new algorithm that builds on INK-Estimate but uses unnormalized RLS. As a consequence, the algorithm is simpler, does not need to estimate the effective dimension for normalization, and achieves a space complexity that is only a constant factor worse than exact RLS sampling.