Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limit

TL;DR

This paper proves that Gaussian RBF kernel CKA converges to linear CKA in the large bandwidth limit, with convergence influenced by feature geometry and eccentricity, affecting nonlinearity range.

Contribution

It establishes the theoretical convergence of Gaussian RBF CKA to linear CKA in the large bandwidth limit and analyzes the impact of feature geometry on this behavior.

Findings

CKA converges to linear CKA as bandwidth increases

Representation eccentricity bounds the nonlinearity range

Geometry of features influences convergence onset

Abstract

We prove that Centered Kernel Alignment (CKA) based on a Gaussian RBF kernel converges to linear CKA in the large-bandwidth limit. We show that convergence onset is sensitive to the geometry of the feature representations, and that representation eccentricity bounds the range of bandwidths for which Gaussian CKA behaves nonlinearly.