VM14K: First Vietnamese Medical Benchmark

TL;DR

This paper introduces VM14K, the first Vietnamese medical benchmark with 14,000 questions across 34 specialties, designed to evaluate language models' medical understanding in Vietnamese.

Contribution

It presents a scalable approach to create non-English medical benchmarks, including data collection, expert annotation, and open-source tools, for the first time in Vietnamese.

Findings

Benchmark covers 34 specialties and 4 difficulty levels.

Includes curated questions from medical exams and clinical records.

Supports evaluation of language models' medical knowledge in Vietnamese.

Abstract

Medical benchmarks are indispensable for evaluating the capabilities of language models in healthcare for non-English-speaking communities,therefore help ensuring the quality of real-life applications. However, not every community has sufficient resources and standardized methods to effectively build and design such benchmark, and available non-English medical data is normally fragmented and difficult to verify. We developed an approach to tackle this problem and applied it to create the first Vietnamese medical question benchmark, featuring 14,000 multiple-choice questions across 34 medical specialties. Our benchmark was constructed using various verifiable sources, including carefully curated medical exams and clinical records, and eventually annotated by medical experts. The benchmark includes four difficulty levels, ranging from foundational biological knowledge commonly found in textbooks to typical clinical case studies that require advanced reasoning. This design enables assessment of both the breadth and depth of language models' medical understanding in the target language thanks to its extensive coverage and in-depth subject-specific expertise. We release the benchmark in three parts: a sample public set (4k questions), a full public set (10k questions), and a private set (2k questions) used for leaderboard evaluation. Each set contains all medical subfields and difficulty levels. Our approach is scalable to other languages, and we open-source our data construction pipeline to support the development of future multilingual benchmarks in the medical domain.