Unveiling the Competitive Dynamics: A Comparative Evaluation of American and Chinese LLMs

TL;DR

This paper compares American and Chinese Large Language Models across languages and tasks, highlighting performance disparities and emphasizing the need for culturally nuanced development and international collaboration.

Contribution

It introduces a comprehensive evaluation framework for LLMs in multiple languages and assesses 16 models, revealing performance gaps and strategic insights.

Findings

GPT 4-Turbo leads in English contexts

Ernie-Bot 4 excels in Chinese contexts

Performance varies significantly across languages and tasks

Abstract

The strategic significance of Large Language Models (LLMs) in economic expansion, innovation, societal development, and national security has been increasingly recognized since the advent of ChatGPT. This study provides a comprehensive comparative evaluation of American and Chinese LLMs in both English and Chinese contexts. We proposed a comprehensive evaluation framework that encompasses natural language proficiency, disciplinary expertise, and safety and responsibility, and systematically assessed 16 prominent models from the US and China under various operational tasks and scenarios. Our key findings show that GPT 4-Turbo is at the forefront in English contexts, whereas Ernie-Bot 4 stands out in Chinese contexts. The study also highlights disparities in LLM performance across languages and tasks, stressing the necessity for linguistically and culturally nuanced model development. The complementary strengths of American and Chinese LLMs point to the value of Sino-US collaboration in advancing LLM technology. The research presents the current LLM competition landscape and offers valuable insights for policymakers and businesses regarding strategic LLM investments and development. Future work will expand on this framework to include emerging LLM multimodal capabilities and business application assessments.