Sharp convergence rates for Spectral methods via the feature space decomposition method

TL;DR

This paper uses the Feature Space Decomposition method to establish precise convergence rates for spectral methods in linear regression, providing a framework to compare their efficiency and limitations.

Contribution

It introduces a general approach to determine and compare the convergence rates of spectral algorithms in linear regression, including conditions for saturation effects.

Findings

Matching upper and lower bounds for excess risk are derived.

A pre-ordering of spectral methods based on convergence rates is established.

Conditions for the saturation effect and limitations in single-index learning are characterized.

Abstract

In this paper, we apply the Feature Space Decomposition (FSD) method developed in [LS24, GLS25, LSSW26, ALSS26] to obtain, under fairly general conditions, matching upper and lower bounds for the population excess risk of spectral methods in linear regression under the squared loss, for every covariance and every signal. This result enables us, for a given linear regression problem, to define a pre-order on the set of spectral methods according to their convergence rates, thereby characterizing which spectral algorithm is superior for that specific problem. Furthermore, this allows us to generalize the saturation effect proposed in inverse problems and to provide necessary and sufficient conditions for its occurrence. Our method also shows that, under broad conditions, any spectral algorithm cannot overcome the barrier of the information exponent in problems such as single-index learning.