CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context

TL;DR

CAPITU is a benchmark designed to evaluate the instruction-following abilities of Large Language Models in Brazilian Portuguese, using culturally-grounded literary tasks and automatic verification methods.

Contribution

It introduces a novel benchmark with culturally-contextualized, verifiable tasks in Portuguese, including diverse linguistic and structural constraints, and provides comprehensive evaluation of state-of-the-art models.

Findings

High accuracy of reasoning models like GPT-5.2 (98.5%)

Portuguese-specialized models offer cost-efficient performance

Multi-turn evaluation shows significant variation in constraint persistence

Abstract

We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at \$0.13 vs Claude-Haiku-4.5: 73.5% at \$1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese.