UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking

TL;DR

This paper introduces UrduFactBench and UrduFactQA, the first Urdu-specific fact-checking benchmarks, along with UrduFactCheck, a modular framework that improves factual verification for Urdu LLMs through evidence retrieval and translation strategies.

Contribution

The work provides the first annotated benchmarks for Urdu fact-checking and develops a novel fact-checking framework that enhances LLM factuality in Urdu using evidence and translation methods.

Findings

Translation-augmented pipelines outperform monolingual ones.

Persistent challenges remain for open-source Urdu LLMs.

Resources significantly aid Urdu fact verification.

Abstract

The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu. Existing automated fact-checking systems are predominantly developed for English, leaving a significant gap for the more than 200 million Urdu speakers worldwide. In this work, we present UrduFactBench and UrduFactQA, two novel hand-annotated benchmarks designed to enable fact-checking and factual consistency evaluation in Urdu. While UrduFactBench focuses on claim verification, UrduFactQA targets the factuality of LLMs in question answering. These resources, the first of their kind for Urdu, were developed through a multi-stage annotation process involving native Urdu speakers. To complement these benchmarks, we introduce UrduFactCheck, a modular fact-checking framework that incorporates both monolingual and translation-based evidence retrieval strategies to mitigate the scarcity of high-quality Urdu evidence. Leveraging these resources, we conduct an extensive evaluation of twelve LLMs and demonstrate that translation-augmented pipelines consistently enhance performance compared to monolingual ones. Our findings reveal persistent challenges for open-source LLMs in Urdu and underscore the importance of developing targeted resources. All code and data are publicly available at https://github.com/mbzuai-nlp/UrduFactCheck.