Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality

TL;DR

The paper introduces the Conceptual Cultural Index (CCI), a new metric for quantifying cultural specificity of sentences in large language models, enabling better evaluation and control of cultural nuances.

Contribution

The paper proposes the CCI metric that measures cultural specificity at the sentence level using generality estimates, improving interpretability and evaluation of multicultural language models.

Findings

CCI effectively distinguishes culture-specific from general sentences.

CCI outperforms direct LLM scoring with over 10-point AUC improvement.

Validation on 400 sentences confirms CCI's reliability and interpretability.

Abstract

Large language models (LLMs) are increasingly deployed in multicultural settings; however, systematic evaluation of cultural specificity at the sentence level remains underexplored. We propose the Conceptual Cultural Index (CCI), which estimates cultural specificity at the sentence level. CCI is defined as the difference between the generality estimate within the target culture and the average generality estimate across other cultures. This formulation enables users to operationally control the scope of culture via comparison settings and provides interpretability, since the score derives from the underlying generality estimates. We validate CCI on 400 sentences (200 culture-specific and 200 general), and the resulting score distribution exhibits the anticipated pattern: higher for culture-specific sentences and lower for general ones. For binary separability, CCI outperforms direct LLM scoring, yielding more than a 10-point improvement in AUC for models specialized to the target culture. Our code is available at https://github.com/IyatomiLab/CCI .