Enabling On-Device Medical AI Assistants via Input-Driven Saliency Adaptation

TL;DR

This paper presents a domain-specific compression framework for large language models, enabling efficient, real-time medical AI assistants on edge devices by pruning and quantizing models tailored to healthcare data.

Contribution

It introduces a neuron saliency-based pruning method combined with post-training quantization for domain-specific LLM compression, optimized for edge deployment in healthcare.

Findings

Significant model size reduction while maintaining performance on medical benchmarks.

Successful deployment of compressed models on low-power edge devices like Jetson Orin Nano and Raspberry Pi 5.

Achieved real-time, energy-efficient inference suitable for resource-constrained healthcare environments.

Abstract

Large Language Models (LLMs) have significant impact on the healthcare scenarios but remain prohibitively large for deployment in real-time, resource-constrained environments such as edge devices. In this work, we introduce a novel medical assistant system, optimized through our general-purpose compression framework, which tailors Large Language Models (LLMs) for deployment in specialized domains. By measuring neuron saliency on domain-specific data, our method can aggressively prune irrelevant neurons, reducing model size while preserving performance. Following pruning, we apply post-training quantization to further reduce the memory footprint, and evaluate the compressed model across medical benchmarks including MedMCQA, MedQA, and PubMedQA. We also deploy the 50\% compressed Gemma and the 67\% compressed LLaMA3 models on Jetson Orin Nano (18.7W peak) and Raspberry Pi 5 (6.3W peak), achieving real-time, energy-efficient inference under hardware constraints.