Misaligned by Design: Incentive Failures in Machine Learning

TL;DR

This paper reveals that training AI models with asymmetric loss functions to align with human decision trade-offs can sometimes be counterproductive, suggesting alternative approaches for better alignment in high-stakes settings.

Contribution

It introduces an economic model explaining how asymmetric training objectives can misalign machine learning incentives and proposes training models independently of human objectives before adjustment.

Findings

Asymmetric loss functions can reduce learning incentives.

Training models separately from human objectives can improve alignment.

Theoretical framework explains incentive misalignments in AI training.

Abstract

The cost of error in many high-stakes settings is asymmetric: misdiagnosing pneumonia when absent is an inconvenience, but failing to detect it when present can be life-threatening. Because of this, artificial intelligence (AI) models used to assist such decisions are frequently trained with asymmetric loss functions that incorporate human decision-makers' trade-offs between false positives and false negatives. In two focal applications, we show that this standard alignment practice can backfire. In both cases, it would be better to train the machine learning model with a loss function that ignores the human's objective and then adjust predictions ex post according to that objective. We rationalize this result using an economic model of incentive design with endogenous information acquisition. The key insight from our theoretical framework is that machine classifiers perform not one but two incentivized tasks: choosing how to classify and learning how to classify. We show that while the adjustments engineers use correctly incentivize choosing, they can simultaneously reduce the incentives to learn. Our formal treatment of the problem reveals that methods embraced for their intuitive appeal can in fact misalign human and machine objectives in predictable ways.