Invariant Kernels: Rank Stabilization and Generalization Across Dimensions

TL;DR

This paper investigates how symmetry in high-dimensional data influences the rank of invariant kernels, revealing that symmetry can significantly reduce rank and improve learning efficiency across different data dimensions.

Contribution

The study provides explicit rank formulas for invariant polynomial kernels and demonstrates their impact on minimax optimal regression in symmetric data settings.

Findings

Symmetry reduces kernel matrix rank, making it independent of data dimension.

Invariant polynomial kernels enable dimension-independent learning.

Numerical experiments confirm theoretical rank reductions and optimality results.

Abstract

Symmetry arises often when learning from high dimensional data. For example, data sets consisting of point clouds, graphs, and unordered sets appear routinely in contemporary applications, and exhibit rich underlying symmetries. Understanding the benefits of symmetry on the statistical and numerical efficiency of learning algorithms is an active area of research. In this work, we show that symmetry has a pronounced impact on the rank of kernel matrices. Specifically, we compute the rank of a polynomial kernel of fixed degree that is invariant under various groups acting independently on its two arguments. In concrete circumstances, including the three aforementioned examples, symmetry dramatically decreases the rank making it independent of the data dimension. In such settings, we show that a simple regression procedure is minimax optimal for estimating an invariant polynomial from finitely many samples drawn across different dimensions. We complete the paper with numerical experiments that illustrate our findings.