Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs

TL;DR

This study examines how legacy procurement practices in U.S. cities influence AI adoption, highlighting challenges and opportunities for reform to prevent algorithmic harms in public sector AI procurement.

Contribution

First empirical analysis of how U.S. cities' procurement laws and norms shape AI decision-making and influence efforts to address algorithmic harms.

Findings

Legacy procurement practices determine AI purchasing decisions.

Cities are taking actions to adapt procurement to address harms.

Identified challenges in current procurement reforms.

Abstract

Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. In this paper, we conduct the first empirical study of how United States cities' procurement practices shape critical decisions surrounding public sector AI. We conduct semi-structured interviews with 19 city employees who oversee AI procurement across 7 U.S. cities. We found that cities' legacy procurement practices, which are shaped by decades-old laws and norms, establish infrastructure that determines which AI is purchased, and which actors hold decision-making power over procured AI. We characterize the emerging actions cities have taken to adapt their purchasing practices to address algorithmic harms. From employees' reflections on real-world AI procurements, we identify three key challenges that motivate but are not fully addressed by existing AI procurement reform initiatives. Based on these findings, we discuss implications and opportunities for the FAccT community to support cities in foreseeing and preventing AI harms throughout the public procurement processes.