Learning Expressive Random Feature Models via Parametrized Activations

TL;DR

This paper introduces RFLAF, a novel random feature model with learnable activation functions that significantly expand the function space and improve performance, demonstrated through theoretical analysis and empirical experiments.

Contribution

It proposes a new model with parameterized activation functions within the random feature framework, enhancing expressivity and efficiency over fixed activation RF models.

Findings

RFLAF with RBFs and splines outperform fixed RF models.

RBF-based RFLAF achieves 3 times faster computation than spline-based RFLAF.

Unfreezing parameters in neural networks validates the expressivity of learnable activations.

Abstract

Random feature (RF) method is a powerful kernel approximation technique, but is typically equipped with fixed activation functions, limiting its adaptability across diverse tasks. To overcome this limitation, we introduce the Random Feature Model with Learnable Activation Functions (RFLAF), a novel statistical model that parameterizes activation functions as weighted sums of basis functions within the random feature framework. Examples of basis functions include radial basis functions, spline functions, polynomials, and so forth. For theoretical results, we consider RBFs as representative basis functions. We start with a single RBF as the activation, and then extend the results to multiple RBFs, demonstrating that RF models with learnable activation component largely expand the represented function space. We provide estimates on the required number of samples and random features to achieve low excess risks. For experiments, we test RFLAF with three types of bases: radial basis functions, spline functions and polynomials. Experimental results show that RFLAFs with RBFs and splines consistently outperform other RF models, where RBFs show 3 times faster computational efficiency than splines. We then unfreeze the first-layer parameters and retrain the models, validating the expressivity advantage of learnable activation components on regular two-layer neural networks. Our work provides a deeper understanding of the component of learnable activation functions within modern neural network architectures.