EXECUTE: A Multilingual Benchmark for LLM Token Understanding

TL;DR

EXECUTE is a multilingual benchmark that evaluates large language models' understanding of characters across diverse languages and scripts, revealing varied challenges beyond English character comprehension.

Contribution

It extends the CUTE benchmark to multiple languages, introduces a flexible framework, and provides insights into language-specific token understanding issues in LLMs.

Findings

Languages differ in token processing challenges.

Some languages exhibit word-level issues, others do not.

Sub-character tasks reveal varying model understanding across scripts.

Abstract

The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components.