HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

TL;DR

This paper introduces HealthQA-BR, a comprehensive Portuguese healthcare benchmark revealing significant knowledge gaps in large language models across various health professions, emphasizing the need for detailed evaluation beyond overall accuracy.

Contribution

It presents the first large-scale, system-wide healthcare benchmark for Portuguese, assessing multiple health disciplines and exposing systemic knowledge deficiencies in leading LLMs.

Findings

GPT 4.1 achieves 86.6% accuracy overall

Performance varies greatly across specialties, from 98.7% in Ophthalmology to 60.0% in Neurosurgery

Models show systemic 'spiky' knowledge profiles, highlighting safety concerns

Abstract

The evaluation of Large Language Models (LLMs) in healthcare has been dominated by physician-centric, English-language benchmarks, creating a dangerous illusion of competence that ignores the interprofessional nature of patient care. To provide a more holistic and realistic assessment, we introduce HealthQA-BR, the first large-scale, system-wide benchmark for Portuguese-speaking healthcare. Comprising 5,632 questions from Brazil's national licensing and residency exams, it uniquely assesses knowledge not only in medicine and its specialties but also in nursing, dentistry, psychology, social work, and other allied health professions. We conducted a rigorous zero-shot evaluation of over 20 leading LLMs. Our results reveal that while state-of-the-art models like GPT 4.1 achieve high overall accuracy (86.6%), this top-line score masks alarming, previously unmeasured deficiencies. A granular analysis shows performance plummets from near-perfect in specialties like Ophthalmology (98.7%) to barely passing in Neurosurgery (60.0%) and, most notably, Social Work (68.4%). This "spiky" knowledge profile is a systemic issue observed across all models, demonstrating that high-level scores are insufficient for safety validation. By publicly releasing HealthQA-BR and our evaluation suite, we provide a crucial tool to move beyond single-score evaluations and toward a more honest, granular audit of AI readiness for the entire healthcare team.