AutoDataset: A Lightweight System for Continuous Dataset Discovery and Search

TL;DR

AutoDataset is an automated, real-time system that continuously monitors arXiv to discover, extract, and index new datasets from research papers, significantly improving dataset discovery speed and coverage.

Contribution

We introduce AutoDataset, a lightweight, automated pipeline for real-time dataset discovery from research papers, combining classification, PDF parsing, URL extraction, and semantic search.

Findings

Achieves 0.94 F1 score in dataset paper classification

Reduces dataset discovery time by up to 80%

Enables low-latency natural language search for datasets

Abstract

The continuous expansion of task-specific datasets has become a major driver of progress in machine learning. However, discovering newly released datasets remains difficult, as existing platforms largely depend on manual curation or community submissions, leading to limited coverage and substantial delays. To address this challenge, we introduce AutoDataset, a lightweight, automated system for real-time dataset discovery and retrieval. AutoDataset adopts a paper-first approach by continuously monitoring arXiv to detect and index datasets directly from newly published research. The system operates through a low-overhead multi-stage pipeline. First, a lightweight classifier rapidly filters titles and abstracts to identify papers releasing datasets, achieving an F1 score of 0.94 with an inference latency of 11 ms. For identified papers, we parse PDFs with GROBID and apply a sentence-level extractor to extract dataset descriptions. Dataset URLs are extracted from the paper text with an automated fallback to LaTeX source analysis when needed. Finally, the structured records are indexed using a dense semantic retriever, enabling low-latency natural language search. We deploy AutoDataset as a live system that continuously ingests new papers and provides up-to-date dataset discovery. In practice, it has been shown to significantly reduce the time required for researchers to locate newly released datasets, improving dataset discovery efficiency by up to 80%.