The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study

TL;DR

This study investigates how metadata like venues, authors, and references influence scientific literature tagging across 19 fields using various classifiers, revealing consistent and field-specific effects of metadata on tagging performance.

Contribution

It systematically analyzes metadata's impact on literature tagging across multiple scientific fields and models, extending beyond prior studies limited to specific disciplines.

Findings

Venues consistently improve tagging accuracy across fields

Metadata effects vary between fields, showing unique patterns

Pre-trained language models benefit more from metadata than simpler models

Abstract

Due to the exponential growth of scientific publications on the Web, there is a pressing need to tag each paper with fine-grained topics so that researchers can track their interested fields of study rather than drowning in the whole literature. Scientific literature tagging is beyond a pure multi-label text classification task because papers on the Web are prevalently accompanied by metadata information such as venues, authors, and references, which may serve as additional signals to infer relevant tags. Although there have been studies making use of metadata in academic paper classification, their focus is often restricted to one or two scientific fields (e.g., computer science and biomedicine) and to one specific model. In this work, we systematically study the effect of metadata on scientific literature tagging across 19 fields. We select three representative multi-label classifiers (i.e., a bag-of-words model, a sequence-based model, and a pre-trained language model) and explore their performance change in scientific literature tagging when metadata are fed to the classifiers as additional features. We observe some ubiquitous patterns of metadata's effects across all fields (e.g., venues are consistently beneficial to paper tagging in almost all cases), as well as some unique patterns in fields other than computer science and biomedicine, which are not explored in previous studies.