The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

TL;DR

This paper introduces the Text Aphasia Battery (TAB), a benchmark adapted from clinical assessments to evaluate aphasic-like language deficits in large language models, with validated automated scoring for large-scale analysis.

Contribution

The paper presents the TAB benchmark, including its design, scoring, and validation, enabling systematic assessment of language deficits in LLMs using a clinically-grounded framework.

Findings

TAB achieves reliability comparable to expert human raters.

Automated evaluation protocol is scalable for large-scale use.

Provides a new tool for analyzing language deficits in artificial systems.

Abstract

Large language models (LLMs) have emerged as a candidate "model organism" for human language, offering an unprecedented opportunity to study the computational basis of linguistic disorders like aphasia. However, traditional clinical assessments are ill-suited for LLMs, as they presuppose human-like pragmatic pressures and probe cognitive processes not inherent to artificial architectures. We introduce the Text Aphasia Battery (TAB), a text-only benchmark adapted from the Quick Aphasia Battery (QAB) to assess aphasic-like deficits in LLMs. The TAB comprises four subtests: Connected Text, Word Comprehension, Sentence Comprehension, and Repetition. This paper details the TAB's design, subtests, and scoring criteria. To facilitate large-scale use, we validate an automated evaluation protocol using Gemini 2.5 Flash, which achieves reliability comparable to expert human raters (prevalence-weighted Cohen's kappa = 0.255 for model--consensus agreement vs. 0.286 for human--human agreement). We release TAB as a clinically-grounded, scalable framework for analyzing language deficits in artificial systems.