IMB: An Italian Medical Benchmark for Question Answering

TL;DR

This paper introduces two extensive Italian medical question answering benchmarks, demonstrating that domain-specific models and retrieval techniques outperform larger general models in medical QA tasks.

Contribution

The paper presents IMB-QA and IMB-MCQA benchmarks for Italian medical QA, and shows how specialized models and retrieval methods enhance performance over larger general models.

Findings

Domain-specific models outperform larger general models.

Retrieval augmented generation improves answer accuracy.

Datasets and evaluation tools are publicly available.

Abstract

Online medical forums have long served as vital platforms where patients seek professional healthcare advice, generating vast amounts of valuable knowledge. However, the informal nature and linguistic complexity of forum interactions pose significant challenges for automated question answering systems, especially when dealing with non-English languages. We present two comprehensive Italian medical benchmarks: \textbf{IMB-QA}, containing 782,644 patient-doctor conversations from 77 medical categories, and \textbf{IMB-MCQA}, comprising 25,862 multiple-choice questions from medical specialty examinations. We demonstrate how Large Language Models (LLMs) can be leveraged to improve the clarity and consistency of medical forum data while retaining their original meaning and conversational style, and compare a variety of LLM architectures on both open and multiple-choice question answering tasks. Our experiments with Retrieval Augmented Generation (RAG) and domain-specific fine-tuning reveal that specialized adaptation strategies can outperform larger, general-purpose models in medical question answering tasks. These findings suggest that effective medical AI systems may benefit more from domain expertise and efficient information retrieval than from increased model scale. We release both datasets and evaluation frameworks in our GitHub repository to support further research on multilingual medical question answering: https://github.com/PRAISELab-PicusLab/IMB.