GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data

TL;DR

GRANDE introduces a gradient-based approach for training decision tree ensembles tailored for tabular data, combining axis-aligned splits with end-to-end optimization to outperform existing methods.

Contribution

It proposes a novel gradient-based decision tree ensemble method using dense representations and straight-through backpropagation, specifically designed for tabular data.

Findings

Outperforms existing gradient-boosting frameworks on most datasets

Effectively learns simple and complex relations within a single model

Demonstrates strong results on 19 classification datasets

Abstract

Despite the success of deep learning for text and image data, tree-based ensemble models are still state-of-the-art for machine learning with heterogeneous tabular data. However, there is a significant need for tabular-specific gradient-based methods due to their high flexibility. In this paper, we propose $\text{GRANDE}$, $\text{GRA}$die$\text{N}$t-Based $\text{D}$ecision Tree $\text{E}$nsembles, a novel approach for learning hard, axis-aligned decision tree ensembles using end-to-end gradient descent. GRANDE is based on a dense representation of tree ensembles, which affords to use backpropagation with a straight-through operator to jointly optimize all model parameters. Our method combines axis-aligned splits, which is a useful inductive bias for tabular data, with the flexibility of gradient-based optimization. Furthermore, we introduce an advanced instance-wise weighting that facilitates learning representations for both, simple and complex relations, within a single model. We conducted an extensive evaluation on a predefined benchmark with 19 classification datasets and demonstrate that our method outperforms existing gradient-boosting and deep learning frameworks on most datasets. The method is available under: https://github.com/s-marton/GRANDE