Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning

TL;DR

This paper benchmarks on-device large language models for clinical reasoning, assessing their accuracy, efficiency, and feasibility on mobile devices, highlighting the potential and challenges of deploying AI in healthcare settings.

Contribution

It provides a comprehensive performance analysis of publicly available on-device LLMs in medical contexts, emphasizing practical deployment considerations.

Findings

Compact models like Phi-3 Mini balance speed and accuracy.

Medically fine-tuned models like Med42 and Aloe achieve higher accuracy.

Older devices can run LLMs, but memory constraints are significant.

Abstract

The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning.