Principled Frameworks for Evaluating Ethics in NLP Systems

TL;DR

This paper emphasizes the importance of understanding ethical frameworks, particularly deontological ethics, as a foundation for evaluating fairness and justice in NLP systems, proposing a research agenda aligned with these principles.

Contribution

It introduces the need to analyze ethical frameworks in NLP evaluation and outlines a deontological approach as a new perspective for future research.

Findings

Highlights the focus on data and modeling interventions in current ethics discussions

Proposes deontological ethics as a foundational framework for evaluating NLP fairness

Suggests a research agenda based on ethical principles rather than just technical metrics

Abstract

We critique recent work on ethics in natural language processing. Those discussions have focused on data collection, experimental design, and interventions in modeling. But we argue that we ought to first understand the frameworks of ethics that are being used to evaluate the fairness and justice of algorithmic systems. Here, we begin that discussion by outlining deontological ethics, and envision a research agenda prioritized by it.