EtiCor++: Towards Understanding Etiquettical Bias in LLMs

TL;DR

EtiCor++ is a new corpus and evaluation framework designed to assess and understand regional etiquette biases in large language models, highlighting their cultural sensitivities and biases.

Contribution

This paper introduces EtiCor++, a comprehensive etiquette corpus and evaluation tasks for measuring regional etiquette knowledge and biases in LLMs.

Findings

LLMs exhibit inherent biases towards certain regions' etiquettes.

EtiCor++ enables systematic evaluation of cultural sensitivity in LLMs.

The framework highlights the need for more culturally aware language models.

Abstract

In recent years, researchers have started analyzing the cultural sensitivity of LLMs. In this respect, Etiquettes have been an active area of research. Etiquettes are region-specific and are an essential part of the culture of a region; hence, it is imperative to make LLMs sensitive to etiquettes. However, there needs to be more resources in evaluating LLMs for their understanding and bias with regard to etiquettes. In this resource paper, we introduce EtiCor++, a corpus of etiquettes worldwide. We introduce different tasks for evaluating LLMs for knowledge about etiquettes across various regions. Further, we introduce various metrics for measuring bias in LLMs. Extensive experimentation with LLMs shows inherent bias towards certain regions.