Cultural Value Differences of LLMs: Prompt, Language, and Model Size

TL;DR

This study investigates how large language models exhibit cultural values based on prompt language, question order, and model size, revealing that language and size significantly influence cultural expression.

Contribution

It systematically analyzes the impact of prompt language, question order, and model size on the cultural values demonstrated by LLMs, highlighting key influencing factors.

Findings

Prompt language affects cultural value expression.

Model size has a greater impact than model type.

Cultural values are consistent within a single language prompt.

Abstract

Our study aims to identify behavior patterns in cultural values exhibited by large language models (LLMs). The studied variants include question ordering, prompting language, and model size. Our experiments reveal that each tested LLM can efficiently behave with different cultural values. More interestingly: (i) LLMs exhibit relatively consistent cultural values when presented with prompts in a single language. (ii) The prompting language e.g., Chinese or English, can influence the expression of cultural values. The same question can elicit divergent cultural values when the same LLM is queried in a different language. (iii) Differences in sizes of the same model (e.g., Llama2-7B vs 13B vs 70B) have a more significant impact on their demonstrated cultural values than model differences (e.g., Llama2 vs Mixtral). Our experiments reveal that query language and model size of LLM are the main factors resulting in cultural value differences.