Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval

TL;DR

This paper presents a multi-stage training approach for a lightweight bilingual Islamic neural retrieval model using domain adaptation, data augmentation, and large datasets, outperforming monolingual models in retrieval tasks.

Contribution

It introduces a novel multi-stage training process for a lightweight bilingual Islamic LLM tailored for neural passage retrieval, addressing domain-specific challenges.

Findings

Bilingual Islamic LLM outperforms monolingual models in retrieval tasks.

Domain adaptation and multi-stage training improve retrieval performance.

Curated in-domain English dataset enhances model effectiveness.

Abstract

This study examines the use of Natural Language Processing (NLP) technology within the Islamic domain, focusing on developing an Islamic neural retrieval model. By leveraging the robust XLM-R model, the research employs a language reduction technique to create a lightweight bilingual large language model (LLM). Our approach for domain adaptation addresses the unique challenges faced in the Islamic domain, where substantial in-domain corpora exist only in Arabic while limited in other languages, including English. The work utilizes a multi-stage training process for retrieval models, incorporating large retrieval datasets, such as MS MARCO, and smaller, in-domain datasets to improve retrieval performance. Additionally, we have curated an in-domain retrieval dataset in English by employing data augmentation techniques and involving a reliable Islamic source. This approach enhances the domain-specific dataset for retrieval, leading to further performance gains. The findings suggest that combining domain adaptation and a multi-stage training method for the bilingual Islamic neural retrieval model enables it to outperform monolingual models on downstream retrieval tasks.