AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance

TL;DR

This paper examines how government agencies in various countries implement AI procurement checklists, highlighting challenges and proposing improvements to ensure ethical and effective AI deployment in the public sector.

Contribution

It analyzes real-world AI procurement checklists from Canada, Brazil, and Singapore, identifying pitfalls and suggesting enhancements for regulatory effectiveness.

Findings

Checklists face challenges in expertise, risk assessment, and transparency.

Lessons from mature jurisdictions inform better AI procurement practices.

Recommendations aim to improve regulation efficacy and ethical AI use in government.

Abstract

Public sector use of AI has been quietly on the rise for the past decade, but only recently have efforts to regulate it entered the cultural zeitgeist. While simple to articulate, promoting ethical and effective roll outs of AI systems in government is a notoriously elusive task. On the one hand there are hard-to-address pitfalls associated with AI-based tools, including concerns about bias towards marginalized communities, safety, and gameability. On the other, there is pressure not to make it too difficult to adopt AI, especially in the public sector which typically has fewer resources than the private sector$\unicode{x2014}$conserving scarce government resources is often the draw of using AI-based tools in the first place. These tensions create a real risk that procedures built to ensure marginalized groups are not hurt by government use of AI will, in practice, be performative and ineffective. To inform the latest wave of regulatory efforts in the United States, we look to jurisdictions with mature regulations around government AI use. We report on lessons learned by officials in Brazil, Singapore and Canada, who have collectively implemented risk categories, disclosure requirements and assessments into the way they procure AI tools. In particular, we investigate two implemented checklists: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF). We detail three key pitfalls around expertise, risk frameworks and transparency, that can decrease the efficacy of regulations aimed at government AI use and suggest avenues for improvement.