CharacterBench: Benchmarking Character Customization of Large Language Models

TL;DR

CharacterBench is a comprehensive bilingual benchmark with over 22,000 samples designed to evaluate and improve large language models' ability to customize characters in dialogue, addressing limitations of previous benchmarks.

Contribution

We introduce CharacterBench, the largest detailed benchmark for character customization in LLMs, along with CharacterJudge for efficient evaluation, enhancing the assessment of character-based dialogue capabilities.

Findings

CharacterJudge outperforms GPT-4 in evaluation accuracy.

CharacterBench covers 3,956 characters across 25 categories.

Our methods improve LLMs' character customization performance.

Abstract

Character-based dialogue (aka role-playing) enables users to freely customize characters for interaction, which often relies on LLMs, raising the need to evaluate LLMs' character customization capability. However, existing benchmarks fail to ensure a robust evaluation as they often only involve a single character category or evaluate limited dimensions. Moreover, the sparsity of character features in responses makes feature-focused generative evaluation both ineffective and inefficient. To address these issues, we propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters from 25 detailed character categories. We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response. We enable effective and efficient evaluation by crafting tailored queries for each dimension to induce characters' responses related to specific dimensions. Further, we develop CharacterJudge model for cost-effective and stable evaluations. Experiments show its superiority over SOTA automatic judges (e.g., GPT-4) and our benchmark's potential to optimize LLMs' character customization. Our repository is at https://github.com/thu-coai/CharacterBench.