Yankari: A Monolingual Yoruba Dataset

TL;DR

Yankari is a large-scale, high-quality monolingual Yoruba dataset designed to enhance NLP research and applications for this underrepresented West African language, enabling more accurate language models and linguistic studies.

Contribution

This paper introduces Yankari, the first extensive Yoruba dataset with rigorous quality control, filling a critical resource gap in NLP for this language.

Findings

The dataset contains over 30 million tokens from 51,407 documents.

Automated evaluations show high data quality and diversity.

Yankari enables improved NLP models for Yoruba.

Abstract

This paper presents Yankari, a large-scale monolingual dataset for the Yoruba language, aimed at addressing the critical gap in Natural Language Processing (NLP) resources for this important West African language. Despite being spoken by over 30 million people, Yoruba has been severely underrepresented in NLP research and applications. We detail our methodology for creating this dataset, which includes careful source selection, automated quality control, and rigorous data cleaning processes. The Yankari dataset comprises 51,407 documents from 13 diverse sources, totaling over 30 million tokens. Our approach focuses on ethical data collection practices, avoiding problematic sources and addressing issues prevalent in existing datasets. We provide thorough automated evaluations of the dataset, demonstrating its quality compared to existing resources. The Yankari dataset represents a significant advancement in Yoruba language resources, providing a foundation for developing more accurate NLP models, supporting comparative linguistic studies, and contributing to the digital accessibility of the Yoruba language.