MOSAIC: Unveiling the Moral, Social and Individual Dimensions of Large Language Models

TL;DR

MOSAIC is a comprehensive benchmark that evaluates large language models across moral, social, and individual dimensions, addressing limitations of existing methods focused solely on Moral Foundation Theory.

Contribution

We introduce MOSAIC, the first large-scale benchmark integrating moral, social, and individual assessments for LLMs, expanding beyond traditional MFT-based evaluations.

Findings

MOSAIC effectively evaluates diverse ethical dimensions of LLMs.

Existing MFT-based benchmarks are insufficient for comprehensive ethical assessment.

Our validation shows models vary significantly across different moral and social dimensions.

Abstract

Large Language Models (LLMs) are increasingly deployed in sensitive applications including psychological support, healthcare, and high-stakes decision-making. This expansion has motivated growing research into the ethical and moral foundations underlying LLM behavior, raising critical questions about their reliability in ethical reasoning. However, existing studies and benchmarks rely almost exclusively on Moral Foundation Theory (MFT), largely neglecting other relevant dimensions such as social values, personality traits, and individual characteristics that shape human ethical reasoning. To address these limitations, we introduce MOSAIC, the first large-scale benchmark designed to jointly assess the moral, social, and individual characteristics of LLMs. The benchmark comprises nine validated questionnaires drawn from moral philosophy, psychology, and social theory, alongside four platform-based games designed to probe morally ambiguous scenarios. In total, MOSAIC includes over 600 curated questions and scenarios, released as a ready-to-use, extensible resource for evaluating the behavioral foundations of LLMs. We validate the benchmark across three models from different families, demonstrating its utility across all assessed dimensions and providing the first empirical evidence that MFT alone is insufficient to comprehensively evaluate complex AI systems' ethical behavior. We publicly release the dataset and our benchmark Python library.