QQ: A Toolkit for Language Identifiers and Metadata

TL;DR

QQ is a Python toolkit that simplifies managing, normalizing, and exploring multilingual language metadata across various identifiers and linguistic attributes, aiding multilingual NLP research.

Contribution

Introduces QwanQwa, a lightweight Python toolkit that unifies language metadata management and facilitates exploration and mapping of diverse language identifiers and attributes.

Findings

Integrates multiple language resources into a single interface.

Provides normalization and mapping between language identifiers.

Enables traversal across linguistic attributes like families and regions.

Abstract

The growing number of languages considered in multilingual NLP, including new datasets and tasks, poses challenges regarding properly and accurately reporting which languages are used and how. For example, datasets often use different language identifiers; some use BCP-47 (e.g. en_Latn), others use ISO 639-1 (en), and more linguistically oriented datasets use Glottocodes (stan1293). Mapping between identifiers is manageable for a few dozen languages, but becomes unscalable when dealing with thousands. We introduce QwanQwa, a light-weight Python toolkit for unified language metadata management. QQ integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes. QQ serves both as (1) a simple "glue" library in multilingual NLP research to make working with many languages easier, and (2) as an intuitive way for exploring languages, such as finding related ones through shared scripts, regions or other metadata.