MultiZebraLogic: A Multilingual Logical Reasoning Benchmark

TL;DR

This paper introduces MultiZebraLogic, a multilingual logical reasoning benchmark with diverse, challenging zebra puzzles designed to evaluate large language models across multiple languages and difficulty levels.

Contribution

It presents a new multilingual dataset of zebra puzzles with varied themes and difficulty, along with code for generating adaptable puzzles for logical reasoning evaluation.

Findings

GPT-4o mini struggles with larger puzzles and red herrings.

Model performance is consistent across languages and themes.

Difficulty does not correlate with specific clue types.

Abstract

Measuring the full abilities of large language models (LLMs) requires benchmarks representing multiple tasks. We aim to create large, high-quality datasets for comparison of logical reasoning skills across several languages and of suitable difficulty for LLMs of various reasoning ability. We explore multiple ways of increasing difficulty. We generate zebra puzzles in multiple languages, themes, sizes and including 14 different clue types and 8 red herring types (uninformative clues). We find puzzle sizes 2x3 and 4x5 are sufficiently challenging for GPT-4o mini (a non-reasoning model) and o3-mini (a reasoning model), respectively. Including 5 red herrings decreases o3-mini puzzle-level accuracy on 4x5 puzzles by 15$\pm$7 %. Scores of o3-mini on 4x5 puzzles are not significantly affected by use of English vs. Danish or the common houses theme vs. the country-specific smoerrebroed theme. We find no correlation between difficulty and the selected clue types. Datasets of 128+1024 puzzles are published as MultiZebraLogic in each of nine Germanic languages for sizes 2x3 and 4x5. We publish code for puzzle generation, designed for adaptablity into more languages and themes.