Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar?

TL;DR

This study compares different field-classification systems used in calculating field-normalized citation scores to determine if the resulting scores are consistent or vary significantly across methods.

Contribution

It provides an empirical comparison of six diverse field-classification systems and their impact on normalized citation scores, highlighting the importance of classification choice.

Findings

Systems based on journal sets and intellectual classifications show moderate agreement.

Citation-relation-based systems also show moderate agreement in two cases.

Semantic similarity-based classification differs significantly from other systems.

Abstract

Usage of field-normalized citation scores is a bibliometric standard. Different methods for field-normalization are in use, but also the choice of field-classification system determines the resulting field-normalized citation scores. Using Web of Science data, we calculated field-normalized citation scores using the same formula but different field-classification systems to answer the question if the resulting scores are different or similar. Six field-classification systems were used: three based on citation relations, one on semantic similarity scores (i.e., a topical relatedness measure), one on journal sets, and one on intellectual classifications. Systems based on journal sets and intellectual classifications agree on at least the moderate level. Two out of the three sets based on citation relations also agree on at least the moderate level. Larger differences were observed for the third data set based on citation relations and semantic similarity scores. The main policy implication is that normalized citation impact scores or rankings based on them should not be compared without deeper knowledge of the classification systems that were used to derive these values or rankings.