From RAG to Agentic: Validating Islamic-Medicine Responses with LLM Agents

TL;DR

This paper introduces Tibbe-AG, a comprehensive evaluation pipeline for Islamic medical knowledge in LLMs, demonstrating that retrieval and self-critique significantly enhance accuracy and cultural sensitivity in medical QA.

Contribution

It develops a novel evaluation framework combining retrieval, self-critique, and agentic judgment to validate culturally grounded medical responses in LLMs.

Findings

Retrieval improves factual accuracy by 13%.

Agentic prompts add an additional 10% accuracy gain.

Blending classical texts with retrieval and self-evaluation enhances reliability and cultural sensitivity.

Abstract

Centuries-old Islamic medical texts like Avicenna's Canon of Medicine and the Prophetic Tibb-e-Nabawi encode a wealth of preventive care, nutrition, and holistic therapies, yet remain inaccessible to many and underutilized in modern AI systems. Existing language-model benchmarks focus narrowly on factual recall or user preference, leaving a gap in validating culturally grounded medical guidance at scale. We propose a unified evaluation pipeline, Tibbe-AG, that aligns 30 carefully curated Prophetic-medicine questions with human-verified remedies and compares three LLMs (LLaMA-3, Mistral-7B, Qwen2-7B) under three configurations: direct generation, retrieval-augmented generation, and a scientific self-critique filter. Each answer is then assessed by a secondary LLM serving as an agentic judge, yielding a single 3C3H quality score. Retrieval improves factual accuracy by 13%, while the agentic prompt adds another 10% improvement through deeper mechanistic insight and safety considerations. Our results demonstrate that blending classical Islamic texts with retrieval and self-evaluation enables reliable, culturally sensitive medical question-answering.