Adaptive Neural Trees

TL;DR

Adaptive Neural Trees (ANTs) integrate deep representation learning with decision tree structures, enabling adaptive, hierarchical models that are efficient and capable of capturing meaningful feature separations.

Contribution

This paper introduces ANTs, a novel framework combining neural networks and decision trees with adaptive growth and backpropagation training.

Findings

Achieves competitive classification and regression performance.

Enables lightweight inference through conditional computation.

Learns meaningful hierarchical feature separations.

Abstract

Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures. We unite the two via adaptive neural trees (ANTs) that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers). We demonstrate that, whilst achieving competitive performance on classification and regression datasets, ANTs benefit from (i) lightweight inference via conditional computation, (ii) hierarchical separation of features useful to the task e.g. learning meaningful class associations, such as separating natural vs. man-made objects, and (iii) a mechanism to adapt the architecture to the size and complexity of the training dataset.