Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models

TL;DR

Open-weight AI models performed as well or better than commercial models in cardiology tests, especially when given access to medical references.

Contribution

Demonstrated that open-weight models with retrieval-augmented generation can outperform proprietary models in cardiovascular medicine assessments.

Findings

The open-weight model DeepSeek R1 outperformed all proprietary models and the human average in cardiology board-style questions.

Retrieval-Augmented Generation improved performance across all models, with the most significant gains for smaller models.

Open-weight models offer a viable, lower-cost alternative to proprietary models for clinical applications due to their transparency and configurability.

Abstract

To evaluate the performance of open-weight and proprietary LLMs, with and without Retrieval-Augmented Generation (RAG), on cardiology board-style questions and benchmark them against the human average. We tested 14 LLMs (6 open-weight, 8 proprietary) on 449 multiple-choice questions from the American College of Cardiology Self-Assessment Program (ACCSAP). Accuracy was measured as percent correct. RAG was implemented using a knowledge base of 123 guideline and textbook documents. The open-weight model DeepSeek R1 achieved the highest accuracy at 86.9% (95% CI: 83.4–89.7%), outperforming proprietary models and the human average of 78%. GPT 4o (80.9%, 95% CI: 77.0–84.2%) and the commercial platform OpenEvidence (81.3%, 95% CI: 77.4–84.7%) demonstrated similar performance. A positive correlation between model size and performance was observed within model families, but across families, substantial variability persisted among models with similar parameter counts. After RAG, all models improved, and open-weight models like Mistral Large 2 (78.0%, 95% CI: 73.9–81.5) performed comparably to proprietary alternatives like GPT 4o. Large language models (LLMs) are increasingly integrated into clinical workflows, yet their performance in cardiovascular medicine remains insufficiently evaluated. Open-weight models can match or exceed proprietary systems in cardiovascular knowledge, with RAG particularly beneficial for smaller models. Given their transparency, configurability, and potential for local deployment, open-weight models, strategically augmented, represent viable, lower-cost alternatives for clinical applications. Open-weight LLMs demonstrate competency in cardiovascular medicine comparable to or exceeding that of proprietary models, with and without RAG depending on the model.

In this work, we set out to understand how today’s artificial intelligence systems perform when tested on the kind of questions cardiologists face during board examinations. We compared a wide range of large language models, including both “open-weight” models and commercial “proprietary” ones, and also tested whether giving the models access to trusted cardiology textbooks and guidelines could improve their answers. We found that the best open-weight model actually outperformed all of the commercial models we tested, even exceeding the average score of practicing cardiologists. When we gave the models access to medical reference material, nearly all of them improved, with the biggest gains seen in the smaller and weaker models. This shows that careful design and support can allow smaller, more accessible systems to reach high levels of accuracy. Our results suggest that open-weight models, which can be used locally without sending sensitive patient information to outside servers, may be a safe and cost-effective alternative to commercial products. This matters because it could make powerful AI tools more widely available across hospitals and clinics, while also reducing risks related to privacy, transparency, and cost.

Figures