Investigating the originality of scientific papers across time and domain: A quantitative analysis

TL;DR

This paper introduces a computational measure called Divergent Semantic Integration (DSI), adapted with SciBERT, to quantify the originality of scientific texts and analyzes its relationship with citations across fields and time.

Contribution

It advances the DSI metric by integrating SciBERT for scientific text analysis and applies it to a large dataset to assess originality's impact on citations.

Findings

DSI correlates with early citations more for papers with fewer authors.

DSI varies significantly across scientific fields.

DSI's correlation with citation counts declines over time.

Abstract

The study of creativity in science has long sought quantitative metrics capable of capturing the originality of the scientific insights contained within articles and other scientific works. In recent years, the field has witnessed a substantial expansion of research activity, enabled by advances in natural language processing and network analysis, and has utilised both macro- and micro-scale approaches with success. However, they often do not examine the text itself for evidence of originality. In this paper, we apply a computational measure correlating with originality from creativity science, Divergent Semantic Integration (DSI), to a set of 51,200 scientific abstracts and titles sourced from the Web of Science. To adapt DSI for application to scientific texts, we advance the original BERT method by incorporating SciBERT (a model trained on scientific corpora) into the computation of DSI. In our study, we observe that DSI plays a more pronounced role in the accrual of early citations for papers with fewer authors, varies substantially across subjects and research fields, and exhibits a declining correlation with citation counts over time. Furthermore, by modelling SciBERT- and BERT-DSI as predictors of the logarithm of 5-year citation counts alongside field, publication year, and the logarithm of author count, we find statistically significant relationships, with adjusted R-squared of 0.103 and 0.101 for BERT-DSI and SciBERT-DSI. Because existing scientometric measures rarely assess the originality expressed in textual content, DSI provides a valuable means of directly quantifying the conceptual originality embedded in scientific writing.