Not just about size - A Study on the Role of Distributed Word Representations in the Analysis of Scientific Publications

TL;DR

This paper investigates how distributed word representations derived from scientific publications can enhance the analysis of scholarly texts, highlighting the trade-offs between domain-specific and general corpora in embedding quality.

Contribution

It provides an experimental comparison of domain-specific and general pre-trained embeddings for scientific text analysis, revealing key factors affecting their effectiveness.

Findings

Domain-specific embeddings effectively capture scientific semantics.

Large general corpora can approximate domain-specific embeddings.

Overlap between knowledge areas influences embedding performance.

Abstract

The emergence of knowledge graphs in the scholarly communication domain and recent advances in artificial intelligence and natural language processing bring us closer to a scenario where intelligent systems can assist scientists over a range of knowledge-intensive tasks. In this paper we present experimental results about the generation of word embeddings from scholarly publications for the intelligent processing of scientific texts extracted from SciGraph. We compare the performance of domain-specific embeddings with existing pre-trained vectors generated from very large and general purpose corpora. Our results suggest that there is a trade-off between corpus specificity and volume. Embeddings from domain-specific scientific corpora effectively capture the semantics of the domain. On the other hand, obtaining comparable results through general corpora can also be achieved, but only in the presence of very large corpora of well formed text. Furthermore, We also show that the degree of overlapping between knowledge areas is directly related to the performance of embeddings in domain evaluation tasks.