Extended Japanese Commonsense Morality Dataset with Masked Token and Label Enhancement

TL;DR

This paper expands the Japanese morality dataset using a novel masking and augmentation method, significantly improving model performance in culturally specific moral reasoning tasks.

Contribution

The paper introduces Masked Token and Label Enhancement (MTLE), a new method for dataset expansion that improves moral reasoning model accuracy in Japanese cultural context.

Findings

eJCM dataset size increased to 31,184 sentences

Model trained on eJCM achieved an F1 score of 0.857

Performance on complex tasks improved from 0.681 to 0.756

Abstract

Rapid advancements in artificial intelligence (AI) have made it crucial to integrate moral reasoning into AI systems. However, existing models and datasets often overlook regional and cultural differences. To address this shortcoming, we have expanded the JCommonsenseMorality (JCM) dataset, the only publicly available dataset focused on Japanese morality. The Extended JCM (eJCM) has grown from the original 13,975 sentences to 31,184 sentences using our proposed sentence expansion method called Masked Token and Label Enhancement (MTLE). MTLE selectively masks important parts of sentences related to moral judgment and replaces them with alternative expressions generated by a large language model (LLM), while re-assigning appropriate labels. The model trained using our eJCM achieved an F1 score of 0.857, higher than the scores for the original JCM (0.837), ChatGPT one-shot classification (0.841), and data augmented using AugGPT, a state-of-the-art augmentation method (0.850). Specifically, in complex moral reasoning tasks unique to Japanese culture, the model trained with eJCM showed a significant improvement in performance (increasing from 0.681 to 0.756) and achieved a performance close to that of GPT-4 Turbo (0.787). These results demonstrate the validity of the eJCM dataset and the importance of developing models and datasets that consider the cultural context.