From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

TL;DR

This paper introduces a Japanese instruction dataset for tuning large language models, demonstrating its effectiveness in improving performance in Japanese tasks and providing resources for further research.

Contribution

The creation of a Japanese instruction dataset and its application to tune LLMs, filling a gap in non-English instruction tuning resources.

Findings

Japanese instruction dataset improves LLM performance

Instruction tuning benefits even small models

Resources are publicly available

Abstract

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.