Ridgeless Regression with Random Features

TL;DR

This paper analyzes the statistical properties of ridgeless regression with random features and stochastic gradient descent, revealing a double-descent error curve and proposing a tunable kernel algorithm to optimize spectral density.

Contribution

It introduces a new analysis of ridgeless regression with random features, connecting theoretical insights with a practical, tunable kernel algorithm.

Findings

Random features error exhibits double-descent behavior

Proposed kernel algorithm optimizes spectral density during training

Bridges interpolation theory and practical algorithm design

Abstract

Recent theoretical studies illustrated that kernel ridgeless regression can guarantee good generalization ability without an explicit regularization. In this paper, we investigate the statistical properties of ridgeless regression with random features and stochastic gradient descent. We explore the effect of factors in the stochastic gradient and random features, respectively. Specifically, random features error exhibits the double-descent curve. Motivated by the theoretical findings, we propose a tunable kernel algorithm that optimizes the spectral density of kernel during training. Our work bridges the interpolation theory and practical algorithm.