Decision Machines: Congruent Decision Trees

TL;DR

This paper introduces Decision Machines, a novel matrix-based representation of decision trees that enhances their optimization and interpretability by embedding Boolean tests into a vector space and exploring their relation to attention mechanisms.

Contribution

It proposes a new matrix-based framework for decision trees, enabling interleaved traversal and linking decision trees with attention mechanisms for improved optimization.

Findings

Matrix representation allows efficient traversal of decision trees.

Embedding Boolean tests into vector space enhances interpretability.

Exploring congruence with attention mechanisms opens new optimization avenues.

Abstract

The decision tree recursively partitions the input space into regions and derives axis-aligned decision boundaries from data. Despite its simplicity and interpretability, decision trees lack parameterized representation, which makes it prone to overfitting and difficult to find the optimal structure. We propose Decision Machines, which embed Boolean tests into a binary vector space and represent the tree structure as a matrices, enabling an interleaved traversal of decision trees through matrix computation. Furthermore, we explore the congruence of decision trees and attention mechanisms, opening new avenues for optimizing decision trees and potentially enhancing their predictive power.