Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction

TL;DR

This paper introduces a domain-agnostic dataset featurization method that uses unsupervised data reconstruction with large language models to extract precise, compact, and human-aligned features, scalable to large datasets.

Contribution

It presents a novel feature extraction approach leveraging LLMs for data reconstruction, offering control over feature granularity and scale, and demonstrating effectiveness in diverse dataset modeling tasks.

Findings

Effective in capturing dataset diversity and effectiveness.

Automates feature discovery aligned with human preferences.

Scales well with larger datasets and more features.

Abstract

Interpreting data is central to modern research. Large language models (LLMs) show promise in providing such natural language interpretations of data, yet simple feature extraction methods such as prompting often fail to produce accurate and versatile descriptions for diverse datasets and lack control over granularity and scale. To address these limitations, we propose a domain-agnostic method for dataset featurization that provides precise control over the number of features extracted while maintaining compact and descriptive representations comparable to human labeling. Our method optimizes the selection of informative binary features by evaluating the ability of an LLM to reconstruct the original data using those features. We demonstrate its effectiveness in dataset modeling tasks and through two case studies: (1) Constructing a feature representation of jailbreak tactics that compactly captures both the effectiveness and diversity of a larger set of human-crafted attacks; and (2) automating the discovery of features that align with human preferences, achieving accuracy and robustness comparable to human-crafted features. Moreover, we show that the pipeline scales effectively, improving as additional features are sampled, making it suitable for large and diverse datasets.