RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models

TL;DR

RoleEval is a bilingual benchmark designed to evaluate large language models' knowledge and reasoning about characters from diverse domains, highlighting differences in cultural and language-specific knowledge.

Contribution

This paper introduces RoleEval, a comprehensive bilingual benchmark with 6,000 questions to assess role knowledge in large language models across multiple languages and domains.

Findings

GPT-4 outperforms others on RoleEval-Global

Chinese LLMs excel on RoleEval-Chinese

Knowledge distribution varies across models and languages

Abstract

The rapid evolution of large language models necessitates effective benchmarks for evaluating their role knowledge, which is essential for establishing connections with the real world and providing more immersive interactions. This paper introduces RoleEval, a bilingual benchmark designed to assess the memorization, utilization, and reasoning capabilities of role knowledge. RoleEval comprises RoleEval-Global (including internationally recognized characters) and RoleEval-Chinese (including characters popular in China), with 6,000 Chinese-English parallel multiple-choice questions focusing on 300 influential people and fictional characters drawn from a variety of domains including celebrities, anime, comics, movies, TV series, games, and fictions. These questions cover basic knowledge and multi-hop reasoning abilities, aiming to systematically probe various aspects such as personal information, relationships, abilities, and experiences of the characters. To maintain high standards, we perform a hybrid quality check process combining both automatic and human verification, ensuring that the questions are diverse, challenging, and discriminative. Our extensive evaluations with RoleEval across various open-source and proprietary large language models, under both the zero- and few-shot settings, reveal insightful findings. Notably, while GPT-4 outperforms other models on RoleEval-Global, Chinese large language models excel on RoleEval-Chinese, highlighting significant knowledge distribution differences. We expect that RoleEval would highlight the significance of assessing role knowledge for large language models across various languages and cultural settings.