Evaluation of Word Embeddings for the Social Sciences

TL;DR

This paper presents the creation and evaluation of social science domain-specific word embeddings, demonstrating their extensive coverage and diversity of semantic relationships compared to general models.

Contribution

It introduces a new social science-specific word embedding model and evaluates its performance against general models across multiple linguistic metrics.

Findings

Domain-specific embeddings cover more social science concepts.

They exhibit greater diversity in semantic neighborhoods.

Semantic relationship coverage is more extensive in domain-specific models.

Abstract

Word embeddings are an essential instrument in many NLP tasks. Most available resources are trained on general language from Web corpora or Wikipedia dumps. However, word embeddings for domain-specific language are rare, in particular for the social science domain. Therefore, in this work, we describe the creation and evaluation of word embedding models based on 37,604 open-access social science research papers. In the evaluation, we compare domain-specific and general language models for (i) language coverage, (ii) diversity, and (iii) semantic relationships. We found that the created domain-specific model, even with a relatively small vocabulary size, covers a large part of social science concepts, their neighborhoods are diverse in comparison to more general models. Across all relation types, we found a more extensive coverage of semantic relationships.