Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems

TL;DR

This paper argues that embracing theoretical inconsistencies among Responsible AI metrics enhances moral representation, informational completeness, and model robustness, challenging the pursuit of strict metric consistency.

Contribution

It introduces a perspective that treats conflicting RAI metrics as beneficial, promoting a pluralistic, comprehensive, and regularized approach to responsible AI development.

Findings

Maintaining diverse metrics captures stakeholder values.

Multiple metrics improve ethical concept representation.

Conflicting objectives enhance model robustness.

Abstract

This position paper argues that the theoretical inconsistency often observed among Responsible AI (RAI) metrics, such as differing fairness definitions or tradeoffs between accuracy and privacy, should be embraced as a valuable feature rather than a flaw to be eliminated. We contend that navigating these inconsistencies, by treating metrics as divergent objectives, yields three key benefits: (1) Normative Pluralism: Maintaining a full suite of potentially contradictory metrics ensures that the diverse moral stances and stakeholder values inherent in RAI are adequately represented. (2) Epistemological Completeness: The use of multiple, sometimes conflicting, metrics allows for a more comprehensive capture of multifaceted ethical concepts, thereby preserving greater informational fidelity about these concepts than any single, simplified definition. (3) Implicit Regularization: Jointly optimizing for theoretically conflicting objectives discourages overfitting to one specific metric, steering models towards solutions with enhanced generalization and robustness under real-world complexities. In contrast, efforts to enforce theoretical consistency by simplifying or pruning metrics risk narrowing this value diversity, losing conceptual depth, and degrading model performance. We therefore advocate for a shift in RAI theory and practice: from getting trapped in inconsistency to characterizing acceptable inconsistency thresholds and elucidating the mechanisms that permit robust, approximated consistency in practice.