Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources

TL;DR

This paper introduces a low-resource Japanese medical large language model based on 7B parameters, achieving comparable or superior performance to larger models, and demonstrates effective cross-lingual transfer for medical question-answering.

Contribution

It presents a novel, efficient Japanese medical LLM that operates with fewer resources and achieves high performance, enabling practical local deployment in healthcare.

Findings

The 7B model matches or exceeds larger medical LLMs in benchmarks.

Fine-tuning on Japanese data improves performance in both Japanese and English.

Cross-lingual transfer enhances medical question-answering accuracy.

Abstract

The recent success of large language models (LLMs) and the scaling law has led to a widespread adoption of larger models. Particularly in the healthcare industry, there is an increasing demand for locally operated LLMs due to security concerns. However, the majority of high quality open-source LLMs have a size of 70B parameters, imposing significant financial burdens on users for GPU preparation and operation. To overcome these issues, we present a medical adaptation based on the recent 7B models, which enables the operation in low computational resources. We compare the performance on medical question-answering benchmarks in two languages (Japanese and English), demonstrating that its scores reach parity with or surpass those of currently existing medical LLMs that are ten times larger. We find that fine-tuning an English-centric base model on Japanese medical dataset improves the score in both language, supporting the effect of cross-lingual knowledge transfer. We hope that this study will alleviate financial challenges, serving as a stepping stone for clinical institutions to practically utilize LLMs locally. Our evaluation code is available at https://github.com/stardust-coder/japanese-lm-med-harness.