Deep Trees for (Un)structured Data: Tractability, Performance, and Interpretability

TL;DR

This paper introduces Generalized Soft Trees (GSTs), a new tree-based method capable of handling unstructured data like images, offering improved performance and interpretability over traditional trees and neural networks.

Contribution

The paper proposes GSTs and the DeepTree algorithm, enabling tractable, high-performance, and interpretable tree models for both tabular and image data, surpassing existing methods.

Findings

GSTs outperform CART, Random Forests, XGBoost on benchmark datasets.

Convolutional Trees excel on CIFAR-10 and Fashion MNIST.

GSTs are more interpretable than deep neural networks.

Abstract

Decision Trees have remained a popular machine learning method for tabular datasets, mainly due to their interpretability. However, they lack the expressiveness needed to handle highly nonlinear or unstructured datasets. Motivated by recent advances in tree-based machine learning (ML) techniques and first-order optimization methods, we introduce Generalized Soft Trees (GSTs), which extend soft decision trees (STs) and are capable of processing images directly. We demonstrate their advantages with respect to tractability, performance, and interpretability. We develop a tractable approach to growing GSTs, given by the DeepTree algorithm, which, in addition to new regularization terms, produces high-quality models with far fewer nodes and greater interpretability than traditional soft trees. We test the performance of our GSTs on benchmark tabular and image datasets, including MIMIC-IV, MNIST, Fashion MNIST, CIFAR-10 and Celeb-A. We show that our approach outperforms other popular tree methods (CART, Random Forests, XGBoost) in almost all of the datasets, with Convolutional Trees having a significant edge in the hardest CIFAR-10 and Fashion MNIST datasets. Finally, we explore the interpretability of our GSTs and find that even the most complex GSTs are considerably more interpretable than deep neural networks. Overall, our approach of Generalized Soft Trees provides a tractable method that is high-performing on (un)structured datasets and preserves interpretability more than traditional deep learning methods.