Noise Audits Improve Moral Foundation Classification

TL;DR

This paper introduces two metrics to identify and remove noisy annotations in moral foundation classification datasets, leading to improved classifier accuracy in detecting moral values in text.

Contribution

It proposes novel entropy and silhouette coefficient metrics for auditing annotation noise, enhancing moral foundation classification performance.

Findings

Removing noisy annotations improves classification accuracy

Metrics effectively identify ambiguous or erroneous labels

Auditing reduces annotation noise impact on models

Abstract

Morality plays an important role in culture, identity, and emotion. Recent advances in natural language processing have shown that it is possible to classify moral values expressed in text at scale. Morality classification relies on human annotators to label the moral expressions in text, which provides training data to achieve state-of-the-art performance. However, these annotations are inherently subjective and some of the instances are hard to classify, resulting in noisy annotations due to error or lack of agreement. The presence of noise in training data harms the classifier's ability to accurately recognize moral foundations from text. We propose two metrics to audit the noise of annotations. The first metric is entropy of instance labels, which is a proxy measure of annotator disagreement about how the instance should be labeled. The second metric is the silhouette coefficient of a label assigned by an annotator to an instance. This metric leverages the idea that instances with the same label should have similar latent representations, and deviations from collective judgments are indicative of errors. Our experiments on three widely used moral foundations datasets show that removing noisy annotations based on the proposed metrics improves classification performance.