Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

TL;DR

This paper proposes using the fast-and-frugal heuristics framework to better understand, study, and teach the use of bibliometrics in research evaluation, emphasizing decision-making under uncertainty.

Contribution

It introduces a novel conceptual framework applying heuristics to analyze bibliometric usage in research assessments, bridging decision science and research evaluation.

Findings

Heuristics can improve decision quality in research evaluation.

The framework aids in understanding how bibliometrics are used under uncertainty.

Empirical examples demonstrate heuristics' applicability outside bibliometrics.

Abstract

While bibliometrics are widely used for research evaluation purposes, a common theoretical framework for conceptually understanding, empirically studying, and effectively teaching its usage is lacking. In this paper, we outline such a framework: the fast-and-frugal heuristics research program, proposed originally in the context of the cognitive and decision sciences, lends itself particularly well for understanding and investigating the usage of bibliometrics in research evaluations. Such evaluations represent judgments under uncertainty in which typically not all possible options, their consequences, and those consequences' probabilities of occurring may be known. In these situations of incomplete information, candidate descriptive and prescriptive models of human behavior are heuristics. Heuristics are simple strategies that, by exploiting the structure of environments, can aid people to make smart decisions. Relying on heuristics does not mean trading off accuracy against effort: while reducing complexity, heuristics can yield better decisions than more information-greedy procedures in many decision environments. The prescriptive power of heuristics is documented in a cross-disciplinary literature, cutting across medicine, crime, business, sports, and other domains. We outline the fast-and-frugal heuristics research program, provide examples of past empirical work on heuristics outside the field of bibliometrics, explain why heuristics may be especially suitable for studying the usage of bibliometrics, and propose a corresponding conceptual framework.