Datasets: A Community Library for Natural Language Processing

TL;DR

Datasets is a community-driven library that standardizes access, documentation, and versioning for over 650 NLP datasets, facilitating research and development in natural language processing.

Contribution

Introduces a comprehensive, community-maintained library that simplifies access and management of NLP datasets, supporting diverse research needs.

Findings

Over 650 datasets included

More than 250 contributors involved

Supported various cross-dataset research projects

Abstract

The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.