Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model

TL;DR

This paper presents a new Persian medical dataset and fine-tuning approach to improve small language models' medical knowledge, achieving better accuracy and passing medical exams in a resource-limited setting.

Contribution

Introduces the first curated Persian medical dataset and demonstrates effective fine-tuning of a small language model for medical question answering.

Findings

Enhanced model accuracy in medical QA tasks

Passed the Iranian Basic Medical Science Entrance Exam

Improved Persian-translated MMLU accuracy by 2.67%

Abstract

The rapid advancement of language models has demonstrated the potential of artificial intelligence in the healthcare industry. However, small language models struggle with specialized domains in low-resource languages like Persian. While numerous medical-domain websites exist in Persian, no curated dataset or corpus has been available making ours the first of its kind. This study introduces a newly curated dataset comprising 20k doctor-patient Q\&A pairs and 60\% of a 90-million-token crawled corpus from medical magazines. Using a parameter-efficient fine-tuning approach, we enhanced the medical knowledge of the baseline model, aya-expanse-8b. Benchmark evaluations demonstrate that the fine-tuned model achieves improved accuracy in medical question answering and successfully passed the Iranian Basic Medical Science Entrance Exam (IBSEE) in September 2023, which the baseline model did not. Additionally, the fine-tuned model improved Persian-translated MMLU accuracy by an average of 2.67\%. This work highlights the potential of leveraging open-access online data to enrich small language models in medical fields, providing a novel solution for Persian medical AI applications suitable for resource-constrained environments. Future research could explore multimodal input to further enhance performance.