From Reflection to Repair: A Scoping Review of Dataset Documentation Tools

TL;DR

This paper systematically reviews dataset documentation tools, revealing persistent conceptualization issues that hinder adoption, and advocates for institutional solutions to improve responsible AI practices.

Contribution

It provides a comprehensive analysis of motivations, conceptualizations, and barriers in dataset documentation tool design, proposing a shift towards institutional approaches.

Findings

Four patterns impede documentation adoption: unclear value, decontextualized design, labor demands, and future integration.

Highlights the need for institutional solutions over individual efforts.

Offers actionable recommendations for the HCI community to support sustainable documentation.

Abstract

Dataset documentation is widely recognized as essential for the responsible development of automated systems. Despite growing efforts to support documentation through different kinds of artifacts, little is known about the motivations shaping documentation tool design or the factors hindering their adoption. We present a systematic review supported by mixed-methods analysis of 59 dataset documentation publications to examine the motivations behind building documentation tools, how authors conceptualize documentation practices, and how these tools connect to existing systems, regulations, and cultural norms. Our analysis shows four persistent patterns in dataset documentation conceptualization that potentially impede adoption and standardization: unclear operationalizations of documentation's value, decontextualized designs, unaddressed labor demands, and a tendency to treat integration as future work. Building on these findings, we propose a shift in Responsible AI tool design toward institutional rather than individual solutions, and outline actions the HCI community can take to enable sustainable documentation practices.