Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

TL;DR

This paper introduces Neural Oblivious Decision Ensembles (NODE), a novel deep learning architecture for tabular data that outperforms gradient boosting decision trees in many tasks, offering a new universal framework.

Contribution

The paper proposes NODE, a deep neural network architecture that combines ensemble of oblivious decision trees with gradient-based training for tabular data.

Findings

NODE outperforms GBDT on most tabular datasets

The architecture enables end-to-end training and hierarchical representation learning

Open-source PyTorch implementation available

Abstract

Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.