Making Sense of Data in the Wild: Data Analysis Automation at Scale

TL;DR

This paper introduces an automated system that uses intelligent agents and retrieval augmented generation to analyze, curate, and index datasets from public repositories, enhancing dataset diversity and utility for machine learning.

Contribution

The paper presents a novel automated approach combining intelligent agents with retrieval augmented generation for large-scale data analysis and curation, improving dataset descriptions and retrieval effectiveness.

Findings

More detailed dataset descriptions generated

Higher hit rates in dataset retrieval

Enhanced diversity in dataset selection

Abstract

As the volume of publicly available data continues to grow, researchers face the challenge of limited diversity in benchmarking machine learning tasks. Although thousands of datasets are available in public repositories, the sheer abundance often complicates the search for suitable data, leaving many valuable datasets underexplored. This situation is further amplified by the fact that, despite longstanding advocacy for improving data curation quality, current solutions remain prohibitively time-consuming and resource-intensive. In this paper, we propose a novel approach that combines intelligent agents with retrieval augmented generation to automate data analysis, dataset curation and indexing at scale. Our system leverages multiple agents to analyze raw, unstructured data across public repositories, generating dataset reports and interactive visual indexes that can be easily explored. We demonstrate that our approach results in more detailed dataset descriptions, higher hit rates and greater diversity in dataset retrieval tasks. Additionally, we show that the dataset reports generated by our method can be leveraged by other machine learning models to improve the performance on specific tasks, such as improving the accuracy and realism of synthetic data generation. By streamlining the process of transforming raw data into machine-learning-ready datasets, our approach enables researchers to better utilize existing data resources.