A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP

TL;DR

This paper introduces a Japanese pharmaceutical domain-specific language model trained on extensive biomedical data, along with three new benchmarks for evaluating pharmaceutical NLP tasks, demonstrating strong performance and highlighting ongoing challenges in consistency reasoning.

Contribution

The work presents a novel Japanese pharmaceutical language model and three comprehensive benchmarks for evaluating pharmaceutical NLP capabilities.

Findings

The model outperforms open-source models on pharmaceutical tasks.

It achieves competitive results with commercial models like GPT-4o.

Cross-sentence consistency remains a challenging area for current models.

Abstract

We present a Japanese domain-specific language model for the pharmaceutical field, developed through continual pretraining on 2 billion Japanese pharmaceutical tokens and 8 billion English biomedical tokens. To enable rigorous evaluation, we introduce three new benchmarks: YakugakuQA, based on national pharmacist licensing exams; NayoseQA, which tests cross-lingual synonym and terminology normalization; and SogoCheck, a novel task designed to assess consistency reasoning between paired statements. We evaluate our model against both open-source medical LLMs and commercial models, including GPT-4o. Results show that our domain-specific model outperforms existing open models and achieves competitive performance with commercial ones, particularly on terminology-heavy and knowledge-based tasks. Interestingly, even GPT-4o performs poorly on SogoCheck, suggesting that cross-sentence consistency reasoning remains an open challenge. Our benchmark suite offers a broader diagnostic lens for pharmaceutical NLP, covering factual recall, lexical variation, and logical consistency. This work demonstrates the feasibility of building practical, secure, and cost-effective language models for Japanese domain-specific applications, and provides reusable evaluation resources for future research in pharmaceutical and healthcare NLP. Our model, codes, and datasets are released at https://github.com/EQUES-Inc/pharma-LLM-eval.