Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor

TL;DR

This paper argues for a broader, responsible conception of rigor in AI that encompasses epistemic, normative, conceptual, reporting, and interpretative aspects beyond traditional methodological rigor.

Contribution

It introduces a comprehensive framework for understanding and practicing broader rigor in AI research, addressing current limitations and fostering responsible AI development.

Findings

Proposes six dimensions of broader rigor in AI

Highlights the importance of epistemic and normative considerations

Provides a framework for dialogue among stakeholders

Abstract

In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about the capabilities of AI systems. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders.