On Feature Scaling of Recursive Feature Machines

TL;DR

This paper investigates the behavior of Recursive Feature Machines (RFMs), revealing a double descent-like pattern in test error when adding noise features, and suggests a connection between RFMs and neural network phenomena.

Contribution

It introduces an analysis of RFMs' feature learning behavior, highlighting a double descent pattern in MSE and proposing a link to neural network dynamics.

Findings

Test MSE shows a decrease-increase-decrease pattern with added noise.

The pattern is consistent across dataset sizes and noise parameters.

RFMs exhibit behaviors similar to neural network double descent phenomena.

Abstract

In this technical report, we explore the behavior of Recursive Feature Machines (RFMs), a type of novel kernel machine that recursively learns features via the average gradient outer product, through a series of experiments on regression datasets. When successively adding random noise features to a dataset, we observe intriguing patterns in the Mean Squared Error (MSE) curves with the test MSE exhibiting a decrease-increase-decrease pattern. This behavior is consistent across different dataset sizes, noise parameters, and target functions. Interestingly, the observed MSE curves show similarities to the "double descent" phenomenon observed in deep neural networks, hinting at new connection between RFMs and neural network behavior. This report lays the groundwork for future research into this peculiar behavior.