Progressing beyond Art Masterpieces or Touristic Clich\'es: how to assess your LLMs for cultural alignment?

TL;DR

This paper reviews current methods for assessing cultural alignment in LLMs, proposes improved dataset design guidelines, and demonstrates their effectiveness through contrastive experiments.

Contribution

It introduces new design guidelines for cultural assessment datasets and shows these improve discrimination between culturally specialized and general models.

Findings

New dataset design yields greater discriminative power.

Contrastive experiments distinguish culturally specialized models.

Proposed guidelines address limitations of existing datasets.

Abstract

Although the cultural (mis)alignment of Large Language Models (LLMs) has attracted increasing attention -- often framed in terms of cultural bias -- until recently there has been limited work on the design and development of datasets for cultural assessment. Here, we review existing approaches to such datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a dataset built according to these principles. We further present a series of contrastive experiments conducted with this dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus.