Analyzing Cultural Representations of Emotions in LLMs through Mixed Emotion Survey

TL;DR

This study examines how large language models represent emotions across different cultures, revealing limited cultural alignment, language influence over speaker origin, and greater similarity among East Asian language responses.

Contribution

It introduces a mixed emotion survey methodology to analyze cultural representations in LLMs and explores the effects of language and speaker origin on responses.

Findings

Models show limited cultural alignment with literature.

Written language impacts LLM responses more than speaker origin.

Responses are more similar among East Asian languages.

Abstract

Large Language Models (LLMs) have gained widespread global adoption, showcasing advanced linguistic capabilities across multiple of languages. There is a growing interest in academia to use these models to simulate and study human behaviors. However, it is crucial to acknowledge that an LLM's proficiency in a specific language might not fully encapsulate the norms and values associated with its culture. Concerns have emerged regarding potential biases towards Anglo-centric cultures and values due to the predominance of Western and US-based training data. This study focuses on analyzing the cultural representations of emotions in LLMs, in the specific case of mixed-emotion situations. Our methodology is based on the studies of Miyamoto et al. (2010), which identified distinctive emotional indicators in Japanese and American human responses. We first administer their mixed emotion survey to five different LLMs and analyze their outputs. Second, we experiment with contextual variables to explore variations in responses considering both language and speaker origin. Thirdly, we expand our investigation to encompass additional East Asian and Western European origin languages to gauge their alignment with their respective cultures, anticipating a closer fit. We find that (1) models have limited alignment with the evidence in the literature; (2) written language has greater effect on LLMs' response than information on participants origin; and (3) LLMs responses were found more similar for East Asian languages than Western European languages.