Common Ground, Diverse Roots: The Difficulty of Classifying Common Examples in Spanish Varieties

TL;DR

This paper investigates the challenge of classifying Spanish language varieties, especially common examples that overlap across varieties, and introduces a new dataset for Cuban Spanish to improve variety identification accuracy.

Contribution

It proposes a method using training dynamics and label confidence to detect common examples and errors, and introduces the first Cuban Spanish variety dataset with annotations.

Findings

Effective detection of common examples improves classification accuracy.

Predicted label confidence enhances model performance in variety identification.

First dataset for Cuban Spanish variety identification with annotations.

Abstract

Variations in languages across geographic regions or cultures are crucial to address to avoid biases in NLP systems designed for culturally sensitive tasks, such as hate speech detection or dialog with conversational agents. In languages such as Spanish, where varieties can significantly overlap, many examples can be valid across them, which we refer to as common examples. Ignoring these examples may cause misclassifications, reducing model accuracy and fairness. Therefore, accounting for these common examples is essential to improve the robustness and representativeness of NLP systems trained on such data. In this work, we address this problem in the context of Spanish varieties. We use training dynamics to automatically detect common examples or errors in existing Spanish datasets. We demonstrate the efficacy of using predicted label confidence for our Datamaps \cite{swayamdipta-etal-2020-dataset} implementation for the identification of hard-to-classify examples, especially common examples, enhancing model performance in variety identification tasks. Additionally, we introduce a Cuban Spanish Variety Identification dataset with common examples annotations developed to facilitate more accurate detection of Cuban and Caribbean Spanish varieties. To our knowledge, this is the first dataset focused on identifying the Cuban, or any other Caribbean, Spanish variety.