Between accurate prediction and poor decision making: the AI/ML gap

TL;DR

This paper highlights that overemphasizing prediction accuracy in AI/ML can lead to poor decision making, emphasizing the need to focus on utility estimation for better real-world outcomes.

Contribution

It introduces a theoretical framework to quantify how utility estimation errors impact decision quality, advocating for a shift towards utility-aware AI methodologies.

Findings

Inaccurate utility assessment can be more detrimental than probability errors.

Theoretical and simulation results demonstrate the importance of utility estimation.

A call for the AI community to prioritize utility-aware approaches.

Abstract

Intelligent agents rely on AI/ML functionalities to predict the consequence of possible actions and optimise the policy. However, the effort of the research community in addressing prediction accuracy has been so intense (and successful) that it created the illusion that the more accurate the learner prediction (or classification) the better would have been the final decision. Now, such an assumption is valid only if the (human or artificial) decision maker has complete knowledge of the utility of the possible actions. This paper argues that AI/ML community has taken so far a too unbalanced approach by devoting excessive attention to the estimation of the state (or target) probability to the detriment of accurate and reliable estimations of the utility. In particular, few evidence exists about the impact of a wrong utility assessment on the resulting expected utility of the decision strategy. This situation is creating a substantial gap between the expectations and the effective impact of AI solutions, as witnessed by recent criticisms and emphasised by the regulatory legislative efforts. This paper aims to study this gap by quantifying the sensitivity of the expected utility to the utility uncertainty and comparing it to the one due to probability estimation. Theoretical and simulated results show that an inaccurate utility assessment may as (and sometimes) more harmful than a poor probability estimation. The final recommendation to the community is then to undertake a focus shift from a pure accuracy-driven (or obsessed) approach to a more utility-aware methodology.