Logics and practices of transparency and opacity in real-world applications of public sector machine learning

TL;DR

This paper explores how public sector actors understand and implement transparency and opacity in machine learning systems, highlighting practical insights for developing socio-technical transparency approaches in real-world applications.

Contribution

It provides empirical insights from interviews with public sector actors on transparency practices, addressing the gap between theoretical transparency and on-the-ground realities.

Findings

Transparency influences trust and decision-making.

Practitioners balance transparency with concerns about gaming.

Opacity can be used strategically to protect decision integrity.

Abstract

Machine learning systems are increasingly used to support public sector decision-making across a variety of sectors. Given concerns around accountability in these domains, and amidst accusations of intentional or unintentional bias, there have been increased calls for transparency of these technologies. Few, however, have considered how logics and practices concerning transparency have been understood by those involved in the machine learning systems already being piloted and deployed in public bodies today. This short paper distils insights about transparency on the ground from interviews with 27 such actors, largely public servants and relevant contractors, across 5 OECD countries. Considering transparency and opacity in relation to trust and buy-in, better decision-making, and the avoidance of gaming, it seeks to provide useful insights for those hoping to develop socio-technical approaches to transparency that might be useful to practitioners on-the-ground. An extended, archival version of this paper is available as Veale M., Van Kleek M., & Binns R. (2018). `Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making' Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI'18), http://doi.org/10.1145/3173574.3174014.