Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training

TL;DR

This paper demonstrates that small Persian medical language models can surpass larger models in reasoning skills by using reinforcement learning and preference optimization, even with limited data.

Contribution

The study introduces a novel training approach combining RLAIF and DPO to improve reasoning in small domain-specific language models in Persian.

Findings

Small models outperform larger ones in medical reasoning tasks.

Reinforcement learning with AI feedback enhances reasoning capabilities.

Limited data can be effectively used to develop domain-specific models.

Abstract

Enhancing reasoning capabilities in small language models is critical for specialized applications such as medical question answering, particularly in underrepresented languages like Persian. In this study, we employ Reinforcement Learning with AI Feedback (RLAIF) and Direct preference optimization (DPO) to improve the reasoning skills of a general-purpose Persian language model. To achieve this, we translated a multiple-choice medical question-answering dataset into Persian and used RLAIF to generate rejected-preferred answer pairs, which are essential for DPO training. By prompting both teacher and student models to produce Chain-of-Thought (CoT) reasoning responses, we compiled a dataset containing correct and incorrect reasoning trajectories. This dataset, comprising 2 million tokens in preferred answers and 2.5 million tokens in rejected ones, was used to train a baseline model, significantly enhancing its medical reasoning capabilities in Persian. Remarkably, the resulting model outperformed its predecessor, gaokerena-V, which was trained on approximately 57 million tokens, despite leveraging a much smaller dataset. These results highlight the efficiency and effectiveness of reasoning-focused training approaches in developing domain-specific language models with limited data availability.