Vanilla Gradient Descent for Oblique Decision Trees

TL;DR

This paper introduces DTSemNet, a novel invertible encoding for oblique decision trees that enables training with standard vanilla gradient descent, resulting in more accurate models and faster training times across classification, regression, and reinforcement learning tasks.

Contribution

DTSemNet provides a new semantically equivalent encoding for hard oblique decision trees, allowing the use of vanilla gradient descent for training, improving accuracy and efficiency over existing differentiable DT methods.

Findings

Oblique DTs trained with DTSemNet outperform similar-sized models in accuracy.

Training time for DTs is significantly reduced using DTSemNet.

DTSemNet can efficiently learn DT policies in reinforcement learning tasks.

Abstract

Decision Trees (DTs) constitute one of the major highly non-linear AI models, valued, e.g., for their efficiency on tabular data. Learning accurate DTs is, however, complicated, especially for oblique DTs, and does take a significant training time. Further, DTs suffer from overfitting, e.g., they proverbially "do not generalize" in regression tasks. Recently, some works proposed ways to make (oblique) DTs differentiable. This enables highly efficient gradient-descent algorithms to be used to learn DTs. It also enables generalizing capabilities by learning regressors at the leaves simultaneously with the decisions in the tree. Prior approaches to making DTs differentiable rely either on probabilistic approximations at the tree's internal nodes (soft DTs) or on approximations in gradient computation at the internal node (quantized gradient descent). In this work, we propose DTSemNet, a novel semantically equivalent and invertible encoding for (hard, oblique) DTs as Neural Networks (NNs), that uses standard vanilla gradient descent. Experiments across various classification and regression benchmarks show that oblique DTs learned using DTSemNet are more accurate than oblique DTs of similar size learned using state-of-the-art techniques. Further, DT training time is significantly reduced. We also experimentally demonstrate that DTSemNet can learn DT policies as efficiently as NN policies in the Reinforcement Learning (RL) setup with physical inputs (dimensions $\leq32$). The code is available at https://github.com/CPS-research-group/dtsemnet.