MedMobile: A mobile-sized language model with clinical capabilities

TL;DR

MedMobile is a compact, 3.8-billion-parameter language model designed for medical applications on mobile devices, achieving state-of-the-art performance with low computational costs and high accessibility.

Contribution

This paper introduces MedMobile, a small yet powerful medical language model optimized for mobile devices, with novel pipeline enhancements improving performance without retrieval augmentation.

Findings

MedMobile scores 75.7% on MedQA, surpassing licensed physician passing threshold.

Achieves state-of-the-art performance among models under 5B parameters.

First small model to pass the USMLE with high accuracy.

Abstract

Language models (LMs) have demonstrated expert-level reasoning and recall abilities in medicine. However, computational costs and privacy concerns are mounting barriers to wide-scale implementation. To address these significant limitations, we introduce a parsimonious adaptation of phi-3-mini, MedMobile, a 3.8 billion parameter LM capable of running on a mobile device, for medical applications. We perform a careful set of pipeline additions and demonstrate that chain of thought, ensembling, and fine-tuning lead to the greatest performance gains, while unexpectedly retrieval augmented generation fails to demonstrate significant improvements. We evaluate the efficiency of our pipeline on the MultiMedQA and MedBullets. We demonstrate that MedMobile scores 75.7% on the MedQA (USMLE), surpassing the passing mark for licensed physicians (~60%) and rivaling scores of models 100 times its size. Across the entirety of the MultiMedQA, MedMobile achieves SOTA performance for models with less than 5B parameters and represents the smallest model to pass the MedQA (USMLE). MedMobile holds promise to democratize access to language models in medicine, bolstering lower compute needs and fast inference speeds. With the ability to combat the biggest barriers to entry for language models in medicine, we hope that MedMobile is a critical step forward in developing clinically relevant language models.