Polish-English medical knowledge transfer: A new benchmark and results

TL;DR

This paper introduces a new Polish medical exam dataset with English translations, benchmarking various LLMs and revealing their strengths and limitations in medical knowledge transfer across languages.

Contribution

It presents a novel Polish medical exam benchmark dataset with English translations and systematically evaluates LLMs' performance on this resource.

Findings

GPT-4o approaches human performance

Models face challenges in cross-lingual translation

Performance varies across medical specialties

Abstract

Large Language Models (LLMs) have demonstrated significant potential in handling specialized tasks, including medical problem-solving. However, most studies predominantly focus on English-language contexts. This study introduces a novel benchmark dataset based on Polish medical licensing and specialization exams (LEK, LDEK, PES) taken by medical doctor candidates and practicing doctors pursuing specialization. The dataset was web-scraped from publicly available resources provided by the Medical Examination Center and the Chief Medical Chamber. It comprises over 24,000 exam questions, including a subset of parallel Polish-English corpora, where the English portion was professionally translated by the examination center for foreign candidates. By creating a structured benchmark from these existing exam questions, we systematically evaluate state-of-the-art LLMs, including general-purpose, domain-specific, and Polish-specific models, and compare their performance against human medical students. Our analysis reveals that while models like GPT-4o achieve near-human performance, significant challenges persist in cross-lingual translation and domain-specific understanding. These findings underscore disparities in model performance across languages and medical specialties, highlighting the limitations and ethical considerations of deploying LLMs in clinical practice.