JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning

TL;DR

This paper demonstrates that LoRA-based instruction-tuning enhances Japanese medical question-answering capabilities of large language models, highlighting the importance of domain-specific adaptation and the potential for local model deployment.

Contribution

It introduces a novel application of LoRA-based instruction-tuning for Japanese medical LLMs, providing a multifaceted evaluation and insights into domain adaptation challenges.

Findings

LoRA-based instruction-tuning improves medical QA performance.

Larger models show more significant domain adaptation effects.

Japanese-centric models still face limitations in domain adaptation.

Abstract

In the ongoing wave of impact driven by large language models (LLMs) like ChatGPT, the adaptation of LLMs to medical domain has emerged as a crucial research frontier. Since mainstream LLMs tend to be designed for general-purpose applications, constructing a medical LLM through domain adaptation is a huge challenge. While instruction-tuning is used to fine-tune some LLMs, its precise roles in domain adaptation remain unknown. Here we show the contribution of LoRA-based instruction-tuning to performance in Japanese medical question-answering tasks. In doing so, we employ a multifaceted evaluation for multiple-choice questions, including scoring based on "Exact match" and "Gestalt distance" in addition to the conventional accuracy. Our findings suggest that LoRA-based instruction-tuning can partially incorporate domain-specific knowledge into LLMs, with larger models demonstrating more pronounced effects. Furthermore, our results underscore the potential of adapting English-centric models for Japanese applications in domain adaptation, while also highlighting the persisting limitations of Japanese-centric models. This initiative represents a pioneering effort in enabling medical institutions to fine-tune and operate models without relying on external services.