The performances of the Chinese and U.S. Large Language Models on the Topic of Chinese Culture

TL;DR

This study compares Chinese and U.S. large language models' performance on Chinese cultural questions, revealing Chinese models generally outperform U.S. models, with differences potentially due to training data and localization strategies.

Contribution

It provides a comparative analysis of Chinese and U.S. LLMs on Chinese culture, highlighting performance disparities and possible underlying causes.

Findings

Chinese models outperform U.S. models on Chinese cultural tasks

Gemini 2.5Pro and GPT-5.1 achieve higher accuracy among U.S. models

Performance differences linked to training data and localization strategies

Abstract

Cultural backgrounds shape individuals' perspectives and approaches to problem-solving. Since the emergence of GPT-1 in 2018, large language models (LLMs) have undergone rapid development. To date, the world's ten leading LLM developers are primarily based in China and the United States. To examine whether LLMs released by Chinese and U.S. developers exhibit cultural differences in Chinese-language settings, we evaluate their performance on questions about Chinese culture. This study adopts a direct-questioning paradigm to evaluate models such as GPT-5.1, DeepSeek-V3.2, Qwen3-Max, and Gemini2.5Pro. We assess their understanding of traditional Chinese culture, including history, literature, poetry, and related domains. Comparative analyses between LLMs developed in China and the U.S. indicate that Chinese models generally outperform their U.S. counterparts on these tasks. Among U.S.-developed models, Gemini 2.5Pro and GPT-5.1 achieve relatively higher accuracy. The observed performance differences may potentially arise from variations in training data distribution, localization strategies, and the degree of emphasis on Chinese cultural content during model development.