Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

TL;DR

This paper introduces OCTree, a novel framework that leverages large language models and decision tree reasoning to improve feature generation in tabular data, surpassing existing automated methods.

Contribution

It presents a new LLM-based feature engineering approach that uses decision trees for iterative feedback, eliminating the need for predefined search spaces and enhancing model performance.

Findings

OCTree outperforms existing automated feature engineering methods.

The framework improves prediction accuracy across diverse benchmarks.

Decision tree reasoning effectively guides feature generation.

Abstract

In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.