Critically Engaged Pragmatism: A Scientific Norm and Social, Pragmatist Epistemology for AI Science Evaluation Tools

TL;DR

This paper advocates for a social, pragmatist approach to evaluating AI science tools, emphasizing critical scrutiny of their purposes and reliability rather than viewing them as objective credibility arbiters.

Contribution

It introduces the norm of Critically Engaged Pragmatism, promoting community-based, purpose-specific evaluation of AI tools in scientific research.

Findings

Highlights risks of misusing AI evaluation tools

Proposes a social epistemology framework for assessment

Encourages critical discursive practices in scientific communities

Abstract

Crises in peer review capacity, study replication, and AI-fabricated science have intensified interest in automated tools for assessing scientific research. However, the scientific community has a history of decontextualizing and repurposing credibility markers in inapt ways. I caution that AI science evaluation tools are particularly prone to these kinds of inference by false ascent due to contestation about the purposes to which they should be put, their portability across purposes, and technical demands that prioritize data set size over epistemic fit. To counter this, I argue for a social, pragmatist epistemology and a newly articulated norm of Critically Engaged Pragmatism to enjoin scientific communities to vigorously scrutinize the purposes and purpose-specific reliability of AI science evaluation tools. Under this framework, AI science evaluation tools are not objective arbiters of scientific credibility, but the object of the kinds of critical discursive practices that ground the credibility of scientific communities.