Optimal Activation Functions for the Random Features Regression Model

TL;DR

This paper identifies optimal activation functions for the Random Features Regression model that minimize test error and sensitivity, revealing scenarios where simple or polynomial-based functions are optimal and analyzing their impact on model properties.

Contribution

It provides a closed-form characterization of activation functions that optimize test error and sensitivity in RFR, extending understanding of their effects on model behavior.

Findings

Optimal AFs include linear, saturated linear, or Hermite polynomial functions.

Using optimal AFs affects the double descent phenomenon in RFR.

Optimal AFs influence the regularization parameter's dependence on noise.

Abstract

The asymptotic mean squared test error and sensitivity of the Random Features Regression model (RFR) have been recently studied. We build on this work and identify in closed-form the family of Activation Functions (AFs) that minimize a combination of the test error and sensitivity of the RFR under different notions of functional parsimony. We find scenarios under which the optimal AFs are linear, saturated linear functions, or expressible in terms of Hermite polynomials. Finally, we show how using optimal AFs impacts well-established properties of the RFR model, such as its double descent curve, and the dependency of its optimal regularization parameter on the observation noise level.