A Systematic Review of NeurIPS Dataset Management Practices

TL;DR

This paper systematically reviews dataset management practices at NeurIPS, highlighting inconsistencies in provenance, hosting, and metadata, and emphasizes the need for standardized data infrastructures.

Contribution

It provides a comprehensive overview of current dataset management practices at NeurIPS, identifying key issues and gaps in provenance, metadata, and version control.

Findings

Provenance is often unclear due to ambiguous curation.

Dataset hosting sites vary widely in metadata support.

There is an urgent need for standardized data management infrastructures.

Abstract

As new machine learning methods demand larger training datasets, researchers and developers face significant challenges in dataset management. Although ethics reviews, documentation, and checklists have been established, it remains uncertain whether consistent dataset management practices exist across the community. This lack of a comprehensive overview hinders our ability to diagnose and address fundamental tensions and ethical issues related to managing large datasets. We present a systematic review of datasets published at the NeurIPS Datasets and Benchmarks track, focusing on four key aspects: provenance, distribution, ethical disclosure, and licensing. Our findings reveal that dataset provenance is often unclear due to ambiguous filtering and curation processes. Additionally, a variety of sites are used for dataset hosting, but only a few offer structured metadata and version control. These inconsistencies underscore the urgent need for standardized data infrastructures for the publication and management of datasets.