Navigating Algorithmic Opacity: Folk Theories and User Agency in Semi-Autonomous Vehicles

TL;DR

This paper explores how semi-autonomous vehicle drivers develop folk theories to interpret opaque AI decisions, highlighting the need for transparency and participatory governance to improve trust and safety.

Contribution

It uncovers drivers' folk theories and proposes a framework for participatory algorithmic governance to enhance transparency and accountability in AV systems.

Findings

Drivers use anthropomorphic metaphors to explain AI behavior.

Drivers lack resources to validate their folk theories.

Opacity issues are prominent in complex, adverse conditions.

Abstract

As semi-autonomous vehicles (AVs) become prevalent, drivers must collaborate with AI systems whose decision-making processes remain opaque. This study examines how drivers of AVs develop folk theories to interpret algorithmic behavior that contradicts their expectations. Through 16 semi-structured interviews with drivers in the United States, we investigate the explanatory frameworks drivers construct to make sense of AI decisions, the strategies they employ when systems behave unexpectedly, and their experiences with control handoffs and feedback mechanisms. Our findings reveal that drivers develop sophisticated folk theories -- often using anthropomorphic metaphors describing systems that ``see,'' ``hesitate,'' or become ``overwhelmed'' -- yet lack informational resources to validate these theories or meaningfully participate in algorithmic governance. We identify contexts where algorithmic opacity manifests acutely, including complex intersections, adverse weather, and rural environments. Current AV designs position drivers as passive data sources rather than epistemic agents, creating accountability gaps that undermine trust and safety. Drawing on critical data studies and algorithmic accountability literature, we propose a framework for participatory algorithmic governance that would provide drivers with transparency into AI decision-making and meaningful channels for contributing to system improvement. This research contributes to understanding how users navigate datafied sociotechnical systems in safety-critical contexts.