"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models

TL;DR

This paper introduces a novel method for zero-shot decision tree induction and embedding using large language models, leveraging their world knowledge to outperform traditional data-driven trees on small datasets.

Contribution

It presents a new approach to generate interpretable decision trees and embeddings directly from LLMs without training data, serving as a knowledge-driven baseline for low-data scenarios.

Findings

Zero-shot decision trees can outperform data-driven trees on small datasets.

Tree-derived embeddings outperform data-driven tree embeddings on average.

The approach harnesses LLMs' world knowledge for tabular data tasks.

Abstract

Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can even surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform better than data-driven tree-based embeddings on average. Our decision tree induction and embedding approaches can therefore serve as new knowledge-driven baselines for data-driven machine learning methods in the low-data regime. Furthermore, they offer ways to harness the rich world knowledge within LLMs for tabular machine learning tasks. Our code and results are available at https://github.com/ml-lab-htw/llm-trees.