A Super-Learner with Large Language Models for Medical Emergency Advising

TL;DR

This paper introduces MEDAS, a super-learner system combining multiple large language models to improve diagnostic accuracy in emergency medicine, surpassing individual models and human performance.

Contribution

The study develops a novel super-learner framework that integrates multiple LLMs with meta-learning to enhance diagnostic accuracy in medical emergencies.

Findings

Super-learner achieves 70% accuracy, higher than individual LLMs.

At least one LLM in the ensemble reaches 85% accuracy.

Meta-learning improves collective diagnostic performance.

Abstract

Medical decision-support and advising systems are critical for emergency physicians to quickly and accurately assess patients' conditions and make diagnosis. Artificial Intelligence (AI) has emerged as a transformative force in healthcare in recent years and Large Language Models (LLMs) have been employed in various fields of medical decision-support systems. We studied responses of a group of different LLMs to real cases in emergency medicine. The results of our study on five most renown LLMs showed significant differences in capabilities of Large Language Models for diagnostics acute diseases in medical emergencies with accuracy ranging between 58% and 65%. This accuracy significantly exceeds the reported accuracy of human doctors. We built a super-learner MEDAS (Medical Emergency Diagnostic Advising System) of five major LLMs - Gemini, Llama, Grok, GPT, and Claude). The super-learner produces higher diagnostic accuracy, 70%, even with a quite basic meta-learner. However, at least one of the integrated LLMs in the same super-learner produces 85% correct diagnoses. The super-learner integrates a cluster of LLMs using a meta-learner capable of learning different capabilities of each LLM to leverage diagnostic accuracy of the model by collective capabilities of all LLMs in the cluster. The results of our study showed that aggregated diagnostic accuracy provided by a meta-learning approach exceeds that of any individual LLM, suggesting that the super-learner can take advantage of the combined knowledge of the medical datasets used to train the group of LLMs.