Retrieval-Augmented Clinical Benchmarking for Contextual Model Testing in Kenyan Primary Care: A Methodology Paper

TL;DR

This paper introduces a methodology for creating a Kenyan-specific clinical benchmark dataset using retrieval-augmented generation to evaluate LLMs in local primary care settings, highlighting performance gaps and ensuring cultural relevance.

Contribution

It presents a novel, guideline-driven framework for developing localized clinical benchmarks and evaluation metrics for LLMs in African healthcare contexts.

Findings

LLMs perform significantly worse on Kenyan medical content compared to US benchmarks.

The dataset includes thousands of question-answer pairs aligned with local guidelines.

New evaluation metrics assess reasoning, safety, and adaptability in clinical scenarios.

Abstract

Large Language Models(LLMs) hold promise for improving healthcare access in low-resource settings, but their effectiveness in African primary care remains underexplored. We present a methodology for creating a benchmark dataset and evaluation framework focused on Kenyan Level 2 and 3 clinical care. Our approach uses retrieval augmented generation (RAG) to ground clinical questions in Kenya's national guidelines, ensuring alignment with local standards. These guidelines were digitized, chunked, and indexed for semantic retrieval. Gemini Flash 2.0 Lite was then prompted with guideline excerpts to generate realistic clinical scenarios, multiple-choice questions, and rationale based answers in English and Swahili. Kenyan physicians co-created and refined the dataset, and a blinded expert review process ensured clinical accuracy, clarity, and cultural appropriateness. The resulting Alama Health QA dataset includes thousands of regulator-aligned question answer pairs across common outpatient conditions. Beyond accuracy, we introduce evaluation metrics that test clinical reasoning, safety, and adaptability such as rare case detection (Needle in the Haystack), stepwise logic (Decision Points), and contextual adaptability. Initial results reveal significant performance gaps when LLMs are applied to localized scenarios, consistent with findings that LLM accuracy is lower on African medical content than on US-based benchmarks. This work offers a replicable model for guideline-driven, dynamic benchmarking to support safe AI deployment in African health systems.