Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

TL;DR

This paper critiques the ERA 2010 research performance metrics used in Australian universities, highlighting statistical issues and proposing the indirect H2 index as a more reliable, objective alternative for impact assessment.

Contribution

It introduces the indirect H2 index as a novel, robust bibliometric indicator that addresses limitations of existing metrics used in research evaluations.

Findings

ERA metrics suffer from statistical issues and bias.

The indirect H2 index is objective, easy to compute, and manipulation-resistant.

The H2 index effectively indicates research impact for evaluation purposes.

Abstract

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.