Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector

TL;DR

This paper discusses the challenges of applying AI decision-making systems in the public sector, emphasizing the need for models that align with complex real-world decision-making and proposing guidance for better modeling practices.

Contribution

It introduces a comprehensive toolkit for AI in public decision-making, focusing on aligning models with policy objectives and addressing key challenges like bias and distribution shifts.

Findings

Standard ML methods often rely on simplifying assumptions.

Counterfactual prediction and policy learning can improve decision outcomes.

External stakeholder input is crucial for model relevance.

Abstract

AI-driven decision-making systems are becoming instrumental in the public sector, with applications spanning areas like criminal justice, social welfare, financial fraud detection, and public health. While these systems offer great potential benefits to institutional decision-making processes, such as improved efficiency and reliability, these systems face the challenge of aligning machine learning (ML) models with the complex realities of public sector decision-making. In this paper, we examine five key challenges where misalignment can occur, including distribution shifts, label bias, the influence of past decision-making on the data side, as well as competing objectives and human-in-the-loop on the model output side. Our findings suggest that standard ML methods often rely on assumptions that do not fully account for these complexities, potentially leading to unreliable and harmful predictions. To address this, we propose a shift in modeling efforts from focusing solely on predictive accuracy to improving decision-making outcomes. We offer guidance for selecting appropriate modeling frameworks, including counterfactual prediction and policy learning, by considering how the model estimand connects to the decision-maker's utility. Additionally, we outline technical methods that address specific challenges within each modeling approach. Finally, we argue for the importance of external input from domain experts and stakeholders to ensure that model assumptions and design choices align with real-world policy objectives, taking a step towards harmonizing AI and public sector objectives.