Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching

TL;DR

This study empirically examines whether field-normalization of citations is necessary by analyzing if field effects persist after accounting for factors like co-authors and pages, using propensity score matching on Web of Science data.

Contribution

It provides evidence that field effects on citation rates remain even after controlling for various factors, supporting the need for field-normalization in bibliometric evaluations.

Findings

Field differences in citation rates are only partially reduced after adjustment.

Field-normalization remains essential as field effects do not fully vanish.

Controlling for FICs alone does not eliminate field effects on citations.

Abstract

Field-normalization of citations is bibliometric standard. Despite the observed differences in citation counts between fields, the question remains how strong fields influence citation rates beyond the effect of attributes or factors possibly influencing citations (FICs). We considered several FICs such as number of pages and number of co-authors in this study. We wondered whether there is a separate field-effect besides other effects (e.g., from numbers of pages and co-authors). To find an answer on the question in this study, we applied inverse-probability of treatment weighting (IPW). Using Web of Science data (a sample of 308,231 articles), we investigated whether mean differences among subject categories in citation rates still remain, even if the subject categories are made comparable in the field-related attributes (e.g., comparable of co-authors, comparable number of pages) by IPW. In a diagnostic step of our statistical analyses, we considered propensity scores as covariates in regression analyses to examine whether the differences between the fields in FICs vanish. The results revealed that the differences did not completely vanish but were strongly reduced. We received similar results when we calculated mean value differences of the fields after IPW representing the causal or unconfounded field effects on citations. However, field differences in citation rates remain. The results point out that field-normalization seems to be a prerequisite for citation analysis and cannot be replaced by the consideration of any set of FICs in citation analyses.