Zero-Shot Decision Tree Construction via Large Language Models

TL;DR

This paper presents a method to construct decision trees using large language models in a zero-shot manner, enabling interpretable models without labeled data, and achieving competitive performance on tabular datasets.

Contribution

It introduces a novel zero-shot decision tree construction algorithm leveraging LLMs for core operations, reducing reliance on labeled data while maintaining interpretability.

Findings

Zero-shot decision trees outperform baseline zero-shot methods.

Achieve competitive accuracy compared to supervised decision trees.

Provide transparent, interpretable models addressing data scarcity.

Abstract

This paper introduces a novel algorithm for constructing decision trees using large language models (LLMs) in a zero-shot manner based on Classification and Regression Trees (CART) principles. Traditional decision tree induction methods rely heavily on labeled data to recursively partition data using criteria such as information gain or the Gini index. In contrast, we propose a method that uses the pre-trained knowledge embedded in LLMs to build decision trees without requiring training data. Our approach leverages LLMs to perform operations essential for decision tree construction, including attribute discretization, probability calculation, and Gini index computation based on the probabilities. We show that these zero-shot decision trees can outperform baseline zero-shot methods and achieve competitive performance compared to supervised data-driven decision trees on tabular datasets. The decision trees constructed via this method provide transparent and interpretable models, addressing data scarcity while preserving interpretability. This work establishes a new baseline in low-data machine learning, offering a principled, knowledge-driven alternative to data-driven tree construction.