Is there agreement on the prestige of scholarly book publishers in the Humanities? DELPHI over survey results

TL;DR

This paper evaluates the consensus on scholarly book publisher prestige in the Humanities by applying a Delphi method to refine survey-based rankings, resulting in a more accepted and conclusive categorization.

Contribution

It introduces a Delphi technique to validate and refine the Scholarly Publishers Indicators ranking, enhancing its acceptance and reliability in Humanities research evaluation.

Findings

Delphi method increased consensus among experts.

High concordance between theoretical and empirical rankings.

Refined categorization more acceptable for evaluation purposes.

Abstract

Despite having an important role supporting assessment processes, criticism towards evaluation systems and the categorizations used are frequent. Considering the acceptance by the scientific community as an essential issue for using rankings or categorizations in research evaluation, the aim of this paper is testing the results of rankings of scholarly book publishers' prestige, Scholarly Publishers Indicators (SPI hereafter). SPI is a public, survey-based ranking of scholarly publishers' prestige (among other indicators). The latest version of the ranking (2014) was based on an expert consultation with a large number of respondents. In order to validate and refine the results for Humanities' fields as proposed by the assessment agencies, a Delphi technique was applied with a panel of randomly selected experts over the initial rankings. The results show an equalizing effect of the technique over the initial rankings as well as a high degree of concordance between its theoretical aim (consensus among experts) and its empirical results (summarized with Gini Index). The resulting categorization is understood as more conclusive and susceptible of being accepted by those under evaluation.