Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond

TL;DR

This paper introduces ADC, an automated, cost-effective method for large-scale dataset creation using LLMs, demonstrated by constructing a 1 million image clothing dataset with high label accuracy.

Contribution

The paper presents ADC, a novel automated dataset construction approach that reduces manual effort, improves label quality, and provides benchmark datasets for noise and bias detection.

Findings

ADC achieves 79% agreement with human annotations.

Reduces label noise from 22.2% to 10.7%.

Provides open-source tools and benchmarks for noisy and biased data learning.

Abstract

Large-scale data collection is essential for developing personalized training data, mitigating the shortage of training data, and fine-tuning specialized models. However, creating high-quality datasets quickly and accurately remains a challenge due to annotation errors, the substantial time and costs associated with human labor. To address these issues, we propose Automatic Dataset Construction (ADC), an innovative methodology that automates dataset creation with negligible cost and high efficiency. Taking the image classification task as a starting point, ADC leverages LLMs for the detailed class design and code generation to collect relevant samples via search engines, significantly reducing the need for manual annotation and speeding up the data generation process. To demonstrate ADC at scale, we construct Clothing-ADC: a dataset of over 1 million images spanning 12 main classes and 12,000 fine-grained subclasses. Our automated curation achieves 79\% agreement with human annotators and reduces label noise from 22.2\% to 10.7\%. Despite these advantages, ADC also encounters real-world challenges such as label errors (label noise) and imbalanced data distributions (label bias). We provide open-source software that incorporates existing methods for label error detection, robust learning under noisy and biased data, ensuring a higher-quality training data and more robust model training procedure. Furthermore, we design three benchmark datasets focused on label noise detection, label noise learning, and class-imbalanced learning. These datasets are vital because there are few existing datasets specifically for label noise detection, despite its importance. Finally, we evaluate the performance of existing popular methods on these datasets, thereby facilitating further research in the field.