The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

TL;DR

The paper introduces the Moral Integrity Corpus (MIC), a large dataset of dialogue pairs annotated with moral judgments, to help understand and evaluate the moral aspects of conversational AI systems.

Contribution

It provides a new resource, MIC, with 38k prompt-reply pairs and 99k Rules of Thumb, for analyzing moral assumptions in dialogue systems and benchmarking their moral integrity.

Findings

Neural models can generate new moral rules for unseen interactions.

Current models struggle with complex moral scenarios.

MIC enables systematic evaluation of moral judgments in dialogue systems.

Abstract

Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user's trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The Moral Integrity Corpus, MIC, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot's reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and language models' implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic