Multilingual Medical Reasoning for Question Answering with Large Language Models

TL;DR

This paper introduces a method to generate multilingual medical reasoning traces from Wikipedia, improving medical question answering performance across languages and setting new state-of-the-art results for 8B-parameter LLMs.

Contribution

It presents a retrieval-augmented generation approach to produce multilingual reasoning traces, extending medical QA datasets, and demonstrating improved performance in multilingual medical question answering.

Findings

Generated 500k reasoning traces in English, Italian, and Spanish.

Enhanced QA performance using traces in both few-shot and fine-tuning settings.

Achieved state-of-the-art results among 8B-parameter LLMs on medical benchmarks.

Abstract

Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.