How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects

TL;DR

This paper demonstrates how adjusted predictions and marginal effects can be used to assess the practical significance of citation impact differences in institutional bibliometrics, using empirical data from German universities.

Contribution

It introduces and illustrates the application of adjusted predictions and marginal effects for evaluating citation impact differences in bibliometric studies.

Findings

Adjusted predictions and marginal effects clarify practical significance.

The method simplifies interpretation of citation impact differences.

Empirical analysis on German universities illustrates the approach.

Abstract

Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are for institutional evaluative bibliometrics. As an illustration, we will calculate a regression model using publications (and citation data) produced by four universities in German-speaking countries from 1980 to 2010. We will show how these predictions and effects can be estimated and plotted, and how this makes it far easier to get a practical feel for the substantive meaning of results in evaluative bibliometric studies. We will focus particularly on Average Adjusted Predictions (AAPs), Average Marginal Effects (AMEs), Adjusted Predictions at Representative Values (APRVs) and Marginal Effects at Representative Values (MERVs).