UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension

TL;DR

This paper introduces UQuAD1.0, a large-scale Urdu question answering dataset created through semi-automated methods, and evaluates Transformer-based models on it, achieving promising results for Urdu MRC.

Contribution

The paper presents the first large-scale Urdu QA dataset, UQuAD1.0, and demonstrates the effectiveness of Transformer models like XLMRoBERTa for Urdu MRC tasks.

Findings

Transformer models outperform rule-based baselines.

XLMRoBERTa achieves an F1 score of 0.66 on UQuAD1.0.

UQuAD1.0 enables future research in Urdu machine reading comprehension.

Abstract

In recent years, low-resource Machine Reading Comprehension (MRC) has made significant progress, with models getting remarkable performance on various language datasets. However, none of these models have been customized for the Urdu language. This work explores the semi-automated creation of the Urdu Question Answering Dataset (UQuAD1.0) by combining machine-translated SQuAD with human-generated samples derived from Wikipedia articles and Urdu RC worksheets from Cambridge O-level books. UQuAD1.0 is a large-scale Urdu dataset intended for extractive machine reading comprehension tasks consisting of 49k question Answers pairs in question, passage, and answer format. In UQuAD1.0, 45000 pairs of QA were generated by machine translation of the original SQuAD1.0 and approximately 4000 pairs via crowdsourcing. In this study, we used two types of MRC models: rule-based baseline and advanced Transformer-based models. However, we have discovered that the latter outperforms the others; thus, we have decided to concentrate solely on Transformer-based architectures. Using XLMRoBERTa and multi-lingual BERT, we acquire an F1 score of 0.66 and 0.63, respectively.