Exploring Large Language Models for Feature Selection: A Data-centric Perspective

TL;DR

This paper investigates how large language models can be used for feature selection, comparing data-driven and text-based approaches, and demonstrates the effectiveness of text-based methods in real-world medical tasks.

Contribution

It categorizes LLM-based feature selection methods and provides experimental evidence of the robustness and potential of text-based approaches across different tasks and models.

Findings

Text-based feature selection methods are effective and robust.

Experiments show promising results in medical applications.

Larger LLMs improve feature selection performance.

Abstract

The rapid advancement of Large Language Models (LLMs) has significantly influenced various domains, leveraging their exceptional few-shot and zero-shot learning capabilities. In this work, we aim to explore and understand the LLMs-based feature selection methods from a data-centric perspective. We begin by categorizing existing feature selection methods with LLMs into two groups: data-driven feature selection which requires numerical values of samples to do statistical inference and text-based feature selection which utilizes prior knowledge of LLMs to do semantical associations using descriptive context. We conduct experiments in both classification and regression tasks with LLMs in various sizes (e.g., GPT-4, ChatGPT and LLaMA-2). Our findings emphasize the effectiveness and robustness of text-based feature selection methods and showcase their potentials using a real-world medical application. We also discuss the challenges and future opportunities in employing LLMs for feature selection, offering insights for further research and development in this emerging field.