Wide Boosting

TL;DR

Wide Boosting enhances traditional Gradient Boosting by inserting a matrix multiplication step, enabling it to better handle multi-dimensional correlated outputs and generate more useful embeddings for downstream tasks.

Contribution

The paper introduces Wide Boosting, a simple modification to Gradient Boosting that improves its ability to model multi-dimensional outputs and produce richer data embeddings.

Findings

Wide Boosting outperforms standard Gradient Boosting on multi-dimensional output tasks.

WB generates more useful embeddings for downstream prediction tasks.

The method is inspired by neural network architectures.

Abstract

Gradient Boosting (GB) is a popular methodology used to solve prediction problems by minimizing a differentiable loss function, $L$. GB performs very well on tabular machine learning (ML) problems; however, as a pure ML solver it lacks the ability to fit models with probabilistic but correlated multi-dimensional outputs, for example, multiple correlated Bernoulli outputs. GB also does not form intermediate abstract data embeddings, one property of Deep Learning that gives greater flexibility and performance on other types of problems. This paper presents a simple adjustment to GB motivated in part by artificial neural networks. Specifically, our adjustment inserts a matrix multiplication between the output of a GB model and the loss, $L$. This allows the output of a GB model to have increased dimension prior to being fed into the loss and is thus ``wider'' than standard GB implementations. We call our method Wide Boosting (WB) and show that WB outperforms GB on mult-dimesional output tasks and that the embeddings generated by WB contain are more useful in downstream prediction tasks than GB output predictions alone.