CUTE: Measuring LLMs' Understanding of Their Tokens

TL;DR

This paper introduces CUTE, a benchmark to evaluate how well large language models understand the spelling and orthographic features of their tokens, revealing their limited ability to utilize this knowledge effectively.

Contribution

The paper presents CUTE, a novel benchmark for assessing LLMs' orthographic understanding, highlighting the gap between token knowledge and practical usage.

Findings

Most LLMs know token spellings

LLMs struggle to manipulate text using orthographic info

Limited generalization of orthographic knowledge

Abstract

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.