Gradient Boosting Neural Networks: GrowNet

TL;DR

This paper introduces GrowNet, a gradient boosting framework using shallow neural networks as weak learners, which outperforms traditional boosting methods across classification, regression, and ranking tasks.

Contribution

It presents a novel gradient boosting approach with neural networks as weak learners, incorporating a fully corrective step to improve over classic decision tree boosting.

Findings

Outperforms state-of-the-art boosting methods on multiple datasets.

Effective across classification, regression, and ranking tasks.

Ablation study highlights the impact of model components.

Abstract

A novel gradient boosting framework is proposed where shallow neural networks are employed as ``weak learners''. General loss functions are considered under this unified framework with specific examples presented for classification, regression, and learning to rank. A fully corrective step is incorporated to remedy the pitfall of greedy function approximation of classic gradient boosting decision tree. The proposed model rendered outperforming results against state-of-the-art boosting methods in all three tasks on multiple datasets. An ablation study is performed to shed light on the effect of each model components and model hyperparameters.