Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)

TL;DR

This paper introduces LACES, a new stereotype dataset for Latin America, and a novel adaptive data collection method that enhances stereotype diversity and evaluation of bias mitigation in NLP models.

Contribution

It presents a culturally-specific stereotype dataset for Latin America and a unified adaptive collection methodology that improves data diversity and collection efficiency.

Findings

LACES contains more unique stereotypes than previous datasets.

Debiasing methods are less effective on LACES, indicating regional bias challenges.

The adaptive collection method streamlines stereotype data gathering.

Abstract

The evaluation of societal biases in NLP models is critically hindered by a geo-cultural gap, This leaves regions such as Latin America severely underserved, making it impossible to adequately assess or mitigate the perpetuation of harmful regional stereotypes in language technologies. This paper presents LACES, a stereotype association dataset, for 15 Latin American countries. This dataset includes 4,789 stereotype associations manually created and annotated by 83 participants. The dataset was developed through targeted community partnerships across Latin America. Additionally, in this paper, we propose a novel adaptive data collection methodology that uniquely integrates the sourcing of new stereotype entries and the validation of existing data within a single, unified workflow. This approach results in a resource with more unique stereotypes than previous static collection methods, enabling a more efficient stereotype collection. The paper further supports the quality of LACES by demonstrating reduced efficacy of debiasing methods on this dataset in comparison to existing popular stereotype benchmarks.