PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian

TL;DR

PARSE is the first comprehensive open-domain reasoning question answering benchmark for Persian, enabling evaluation and development of reasoning-capable language models in this low-resource language.

Contribution

It introduces PARSE, a large-scale Persian reasoning QA dataset built with LLM-based generation, human validation, and multi-stage filtering, filling a critical gap in Persian NLP resources.

Findings

Prompting strategies improve model performance.

Fine-tuning enhances results, especially for Persian-specific models.

PARSE enables fair comparison and practical model adaptation in Persian QA.

Abstract

Reasoning-focused Question Answering (QA) has advanced rapidly with Large Language Models (LLMs), yet high-quality benchmarks for low-resource languages remain scarce. Persian, spoken by roughly 130 million people, lacks a comprehensive open-domain resource for evaluating reasoning-capable QA systems. We introduce PARSE, the first open-domain Persian reasoning QA benchmark, containing 10,800 questions across Boolean, multiple-choice, and factoid formats, with diverse reasoning types, difficulty levels, and answer structures. The benchmark is built via a controlled LLM-based generation pipeline and validated through human evaluation. We also ensure linguistic and factual quality through multi-stage filtering, annotation, and consistency checks. We benchmark multilingual and Persian LLMs under multiple prompting strategies and show that Persian prompts and structured prompting (CoT for Boolean/multiple-choice; few-shot for factoid) improve performance. Fine-tuning further boosts results, especially for Persian-specialized models. These findings highlight how PARSE supports both fair comparison and practical model adaptation. PARSE fills a critical gap in Persian QA research and provides a strong foundation for developing and evaluating reasoning-capable LLMs in low-resource settings.