ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction

TL;DR

ELF-Gym introduces a framework for evaluating how well large language models generate features for tabular data, comparing their performance and similarity to human-crafted features across Kaggle datasets.

Contribution

This paper presents ELF-Gym, a novel evaluation framework that assesses LLM-generated features against expert features using performance impact and semantic similarity metrics.

Findings

LLMs can semantically capture about 56% of expert features in best cases

Implementation-level overlap of LLM features with expert features drops to 13%

LLMs often fail on datasets requiring complex feature engineering

Abstract

Crafting effective features is a crucial yet labor-intensive and domain-specific task within machine learning pipelines. Fortunately, recent advancements in Large Language Models (LLMs) have shown promise in automating various data science tasks, including feature engineering. But despite this potential, evaluations thus far are primarily based on the end performance of a complete ML pipeline, providing limited insight into precisely how LLMs behave relative to human experts in feature engineering. To address this gap, we propose ELF-Gym, a framework for Evaluating LLM-generated Features. We curated a new dataset from historical Kaggle competitions, including 251 "golden" features used by top-performing teams. ELF-Gym then quantitatively evaluates LLM-generated features by measuring their impact on downstream model performance as well as their alignment with expert-crafted features through semantic and functional similarity assessments. This approach provides a more comprehensive evaluation of disparities between LLMs and human experts, while offering valuable insights into specific areas where LLMs may have room for improvement. For example, using ELF-Gym we empirically demonstrate that, in the best-case scenario, LLMs can semantically capture approximately 56% of the golden features, but at the more demanding implementation level this overlap drops to 13%. Moreover, in other cases LLMs may fail completely, particularly on datasets that require complex features, indicating broad potential pathways for improvement.