The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AI

TL;DR

This paper discusses how governments can enhance interconnected post-deployment monitoring of AI systems by leveraging data sharing and impact assessment methods to better understand and mitigate AI-related societal impacts.

Contribution

It introduces a framework for government-led interconnected monitoring of AI, emphasizing data sources and mechanisms inspired by other industries for effective risk management.

Findings

Interconnected monitoring combines model, application, and impact data.

Governments can utilize specific data sources for AI risk assessment.

Effective impact assessments require causal, empirical evidence.

Abstract

Language-based AI systems are diffusing into society, bringing positive and negative impacts. Mitigating negative impacts depends on accurate impact assessments, drawn from an empirical evidence base that makes causal connections between AI usage and impacts. Interconnected post-deployment monitoring combines information about model integration and use, application use, and incidents and impacts. For example, inference time monitoring of chain-of-thought reasoning can be combined with long-term monitoring of sectoral AI diffusion, impacts and incidents. Drawing on information sharing mechanisms in other industries, we highlight example data sources and specific data points that governments could collect to inform AI risk management.