The Collapse of Heterogeneity in Silicon Philosophers

TL;DR

This study reveals that large language models tend to artificially homogenize philosophical opinions, overestimating consensus and losing individual heterogeneity, which impacts their use in alignment and evaluation tasks.

Contribution

The paper demonstrates that language models collapse heterogeneity in philosophical judgments, revealing limitations in their ability to replicate individual differences.

Findings

Language models over-correlate philosophical judgments, creating artificial consensus.

Models implicitly assume domain specialists share similar views.

Fine-tuning impacts the degree of heterogeneity collapse.

Abstract

Silicon samples are increasingly used as a low-cost substitute for human panels and have been shown to reproduce aggregate human opinion with high fidelity. We show that, in the alignment-relevant domain of philosophy, silicon samples systematically collapse heterogeneity. Using data from $N = {277}$ professional philosophers drawn from PhilPeople profiles, we evaluate seven proprietary and open-source large language models on their ability to replicate individual philosophical positions and to preserve cross-question correlation structures across philosophical domains. We find that language models substantially over-correlate philosophical judgments, producing artificial consensus across domains. This collapse is associated in part with specialist effects, whereby models implicitly assume that domain specialists hold highly similar philosophical views. We assess the robustness of these findings by studying the impact of DPO fine-tuning and by validating results against the full PhilPapers 2020 Survey ($N = {1785}$). We conclude by discussing implications for alignment, evaluation, and the use of silicon samples as substitutes for human judgment. The code of this project can be found at https://github.com/stanford-del/silicon-philosophers.