Development and Preliminary Evaluation of a Domain-Specific Large Language Model for Tuberculosis Care in South Africa

TL;DR

This paper describes the development and initial testing of a specialized large language model tailored for tuberculosis care in South Africa, utilizing domain-specific data and advanced fine-tuning techniques.

Contribution

It introduces a novel domain-specific LLM for TB care, fine-tuned with QLoRA and retrieval augmentation, showing improved performance over general models.

Findings

The DS-LLM outperformed the base BioMistral-7B in contextual alignment.

Fine-tuning with QLoRA enhanced the model's medical knowledge.

Retrieval-augmented generation contributed to better contextual understanding.

Abstract

Tuberculosis (TB) is one of the world's deadliest infectious diseases, and in South Africa, it contributes a significant burden to the country's health care system. This paper presents an experimental study on the development of a domain-specific Large Language Model (DS-LLM) for TB care that can help to alleviate the burden on patients and healthcare providers. To achieve this, a literature review was conducted to understand current LLM development strategies, specifically in the medical domain. Thereafter, data were collected from South African TB guidelines, selected TB literature, and existing benchmark medical datasets. We performed LLM fine-tuning by using the Quantised Low-Rank Adaptation (QLoRA) algorithm on a medical LLM (BioMistral-7B), and also implemented Retrieval-Augmented Generation using GraphRAG. The developed DS-LLM was evaluated against the base BioMistral-7B model and a general-purpose LLM using a mix of automated metrics and quantitative ratings. The results show that the DS-LLM had better performance compared to the base model in terms of its contextual alignment (lexical, semantic, and knowledge) for TB care in South Africa.