Effects of Cross-lingual Evidence in Multilingual Medical Question Answering

TL;DR

This study explores how different external evidence sources and retrieval strategies impact multilingual medical question answering, revealing that effectiveness varies with language resource availability and model size.

Contribution

It systematically evaluates the effects of curated, web, and LLM explanations across languages and retrieval methods, highlighting nuanced strategies for low-resource settings.

Findings

Larger models perform better in English across evaluations.

Web-retrieved data benefits high-resource languages most.

Combining English and target language retrieval improves low-resource language performance.

Abstract

This paper investigates Multilingual Medical Question Answering across high-resource (English, Spanish, French, Italian) and low-resource (Basque, Kazakh) languages. We evaluate three types of external evidence sources across models of varying size: curated repositories of specialized medical knowledge, web-retrieved content, and explanations from LLM's parametric knowledge. Moreover, we conduct experiments with multilingual, monolingual and cross-lingual retrieval. Our results demonstrate that larger models consistently achieve superior performance in English across baseline evaluations. When incorporating external knowledge, web-retrieved data in English proves most beneficial for high-resource languages. Conversely, for low-resource languages, the most effective strategy combines retrieval in both English and the target language, achieving comparable accuracy to high-resource language results. These findings challenge the assumption that external knowledge systematically improves performance and reveal that effective strategies depend on both the source of language resources and on model scale. Furthermore, specialized medical knowledge sources such as PubMed are limited: while they provide authoritative expert knowledge, they lack adequate multilingual coverage