Deep Regression Ensembles

TL;DR

Deep Regression Ensembles (DRE) are a novel neural network architecture that combines random feature regression with trained output weights, achieving competitive performance with significantly reduced computational costs.

Contribution

The paper introduces DRE, a new neural network design that integrates random feature regression with trained output layers, bridging the gap between traditional DNNs and two-layer networks.

Findings

DRE matches or exceeds state-of-the-art DNN performance on various datasets.

DRE has much lower computational cost compared to traditional DNNs.

DRE's weights are either analytically known or randomly drawn, simplifying training.

Abstract

We introduce a methodology for designing and training deep neural networks (DNN) that we call "Deep Regression Ensembles" (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer of DRE has two components, randomly drawn input weights and output weights trained myopically (as if the final output layer) using linear ridge regression. Within a layer, each neuron uses a different subset of inputs and a different ridge penalty, constituting an ensemble of random feature ridge regressions. Our experiments show that a single DRE architecture is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because DRE neural weights are either known in closed-form or randomly drawn, its computational cost is orders of magnitude smaller than DNN.