JETHICS: Japanese Ethics Understanding Evaluation Dataset
Masashi Takeshita, Rafal Rzepka

TL;DR
This paper introduces JETHICS, a large Japanese ethics dataset for evaluating AI models' understanding of ethical concepts, revealing significant gaps in current LLMs' ethical comprehension.
Contribution
JETHICS is the first extensive Japanese ethics dataset built on established English datasets, enabling cross-linguistic ethical evaluation of AI models.
Findings
GPT-4o scores about 0.7, indicating limited ethics understanding.
The best Japanese LLM scores around 0.5, showing room for improvement.
Current models have substantial gaps in ethics comprehension.
Abstract
In this work, we propose JETHICS, a Japanese dataset for evaluating ethics understanding of AI models. JETHICS contains 78K examples and is built by following the construction methods of the existing English ETHICS dataset. It includes four categories based normative theories and concepts from ethics and political philosophy; and one representing commonsense morality. Our evaluation experiments on non-proprietary large language models (LLMs) and on GPT-4o reveal that even GPT-4o achieves only an average score of about 0.7, while the best-performing Japanese LLM attains around 0.5, indicating a relatively large room for improvement in current LLMs.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsEthics and Social Impacts of AI · Explainable Artificial Intelligence (XAI) · Psychology of Moral and Emotional Judgment
