Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust

TL;DR

This paper presents a lightweight, multilingual IR system for Islamic texts, leveraging domain-specific pre-training, language reduction, and Rust-based multiprocessing to enable efficient, low-resource semantic search.

Contribution

It introduces a novel, resource-efficient multilingual IR system tailored for Islamic literature, combining domain adaptation, language reduction, and Rust multiprocessing for improved performance.

Findings

Superior retrieval performance over general models

Effective domain-specific pre-training and language reduction

Efficient implementation using Rust for low-resource environments

Abstract

The widespread use of large language models (LLMs) has dramatically improved many applications of Natural Language Processing (NLP), including Information Retrieval (IR). However, domains that are not driven by commercial interest often lag behind in benefiting from AI-powered solutions. One such area is religious and heritage corpora. Alongside similar domains, Islamic literature holds significant cultural value and is regularly utilized by scholars and the general public. Navigating this extensive amount of text is challenging, and there is currently no unified resource that allows for easy searching of this data using advanced AI tools. This work focuses on the development of a multilingual non-profit IR system for the Islamic domain. This process brings a few major challenges, such as preparing multilingual domain-specific corpora when data is limited in certain languages, deploying a model on resource-constrained devices, and enabling fast search on a limited budget. By employing methods like continued pre-training for domain adaptation and language reduction to decrease model size, a lightweight multilingual retrieval model was prepared, demonstrating superior performance compared to larger models pre-trained on general domain data. Furthermore, evaluating the proposed architecture that utilizes Rust Language capabilities shows the possibility of implementing efficient semantic search in a low-resource setting.