DOFEN: Deep Oblivious Forest ENsemble

TL;DR

DOFEN introduces a novel deep neural network architecture inspired by oblivious decision trees, achieving state-of-the-art results on tabular data and narrowing the performance gap with traditional tree-based models.

Contribution

The paper proposes DOFEN, a new DNN architecture that constructs relaxed oblivious decision trees and employs a two-level ensembling process for improved tabular data performance.

Findings

DOFEN achieves state-of-the-art results among DNNs on tabular benchmarks.

DOFEN narrows the performance gap between DNNs and tree-based models.

The approach outperforms existing neural network models on diverse tabular datasets.

Abstract

Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. In this paper, we propose DOFEN, short for \textbf{D}eep \textbf{O}blivious \textbf{F}orest \textbf{EN}semble, a novel DNN architecture inspired by oblivious decision trees. DOFEN constructs relaxed oblivious decision trees (rODTs) by randomly combining conditions for each column and further enhances performance with a two-level rODT forest ensembling process. By employing this approach, DOFEN achieves state-of-the-art results among DNNs and further narrows the gap between DNNs and tree-based models on the well-recognized benchmark: Tabular Benchmark \citep{grinsztajn2022tree}, which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: \url{https://github.com/Sinopac-Digital-Technology-Division/DOFEN}.