JETHICS: Japanese Ethics Understanding Evaluation Dataset

TL;DR

This paper introduces JETHICS, a large Japanese ethics dataset for evaluating AI models' understanding of ethical concepts, revealing significant gaps in current LLMs' ethical comprehension.

Contribution

JETHICS is the first extensive Japanese ethics dataset built on established English datasets, enabling cross-linguistic ethical evaluation of AI models.

Findings

GPT-4o scores about 0.7, indicating limited ethics understanding.

The best Japanese LLM scores around 0.5, showing room for improvement.

Current models have substantial gaps in ethics comprehension.

Abstract

In this work, we propose JETHICS, a Japanese dataset for evaluating ethics understanding of AI models. JETHICS contains 78K examples and is built by following the construction methods of the existing English ETHICS dataset. It includes four categories based normative theories and concepts from ethics and political philosophy; and one representing commonsense morality. Our evaluation experiments on non-proprietary large language models (LLMs) and on GPT-4o reveal that even GPT-4o achieves only an average score of about 0.7, while the best-performing Japanese LLM attains around 0.5, indicating a relatively large room for improvement in current LLMs.