Random Feature Representation Boosting

TL;DR

The paper introduces RFRBoost, a boosting-based method for deep residual random feature neural networks that improves performance and offers theoretical guarantees, especially effective on small to medium datasets.

Contribution

It presents a novel boosting approach for RFNNs, providing closed-form solutions for certain loss functions and demonstrating superior empirical performance.

Findings

RFRBoost outperforms RFNNs and MLP ResNets on tabular datasets.

It offers computational benefits over traditional methods.

Theoretical guarantees are established based on boosting theory.

Abstract

We introduce Random Feature Representation Boosting (RFRBoost), a novel method for constructing deep residual random feature neural networks (RFNNs) using boosting theory. RFRBoost uses random features at each layer to learn the functional gradient of the network representation, enhancing performance while preserving the convex optimization benefits of RFNNs. In the case of MSE loss, we obtain closed-form solutions to greedy layer-wise boosting with random features. For general loss functions, we show that fitting random feature residual blocks reduces to solving a quadratically constrained least squares problem. Through extensive numerical experiments on tabular datasets for both regression and classification, we show that RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in the small- to medium-scale regime where RFNNs are typically applied. Moreover, RFRBoost offers substantial computational benefits, and theoretical guarantees stemming from boosting theory.