Quantification and Analysis of Scientific Language Variation Across Research Fields

TL;DR

This paper introduces a neural language model-based method to quantify and analyze linguistic variation across scientific research fields, aiding interdisciplinary understanding and collaboration.

Contribution

It presents a novel computational approach using neural embeddings to measure semantic differences of terms across research domains.

Findings

The model effectively identifies terms with significant semantic change.

It provides a metric to quantify overall linguistic variation.

The approach improves cross-disciplinary data collaboration.

Abstract

Quantifying differences in terminologies from various academic domains has been a longstanding problem yet to be solved. We propose a computational approach for analyzing linguistic variation among scientific research fields by capturing the semantic change of terms based on a neural language model. The model is trained on a large collection of literature in five computer science research fields, for which we obtain field-specific vector representations for key terms, and global vector representations for other words. Several quantitative approaches are introduced to identify the terms whose semantics have drastically changed, or remain unchanged across different research fields. We also propose a metric to quantify the overall linguistic variation of research fields. After quantitative evaluation on human annotated data and qualitative comparison with other methods, we show that our model can improve cross-disciplinary data collaboration by identifying terms that potentially induce confusion during interdisciplinary studies.

Tables

Table 1. TABLE I: Terms with most significant semantic change.

Term	Avg. $Di{s}^k$	Highest $Di{s}^k$ and fields
alignment	0.9215	0.9634 (NLP-Comm)
relations	0.8923	0.9654 (NLP-Arch)
translation	0.8905	0.9773 (NLP-Arch)
mapping	0.8819	0.9817 (NLP-Arch)
embedding	0.8780	0.9533 (NLP-Comm)
embed	0.8675	0.9785 (NLP-Arch)
pattern	0.8670	0.9804 (CV-Arch)
feature	0.8621	0.9667 (NLP-Arch)
grid	0.8509	0.9445 (NLP-CV)
semantic	0.8396	0.9679 (NLP-Arch)

Table 2. TABLE III: Most similar words of some terms in Table  II and II .

Term	Most Similar Words Per Field
	NLP	CV	DBDM	Arch	Comm
alignment	translation bilingual parallel	facial shape localization	matching alignments mapping	sequences binary parallel	movr nlos phasedarray
embedding	embeddings representation vector	embed similarity sparsecoded	embeddings structure space	embed tree bitsequence	embedded visual domain
semantic	semantics syntactic lexical	attributes segmentation representation	ontology semantics web	semantics tracking approach	structured relations indexing
linux	workstation intel unix	intel geforce workstation	intel quadcore operating	kernel software filesystem	kernel userspace software
cryptography	public-key secrets backroom	secure public-key encryption	crypto secure encryption	public-key crypto secure	crypto public-key secure

Table 3. TABLE IV: Comparison with Human Annotations.

Methods	${\rho }_{Pearson}$	nDCG at rank 30	nDCG at rank 50
Separate CBOW–JS	0.492	0.753	0.839
GEODIST–Cosine	-0.488	0.115	0.406
Our Model–Cosine	0.777	0.815	0.858
Our Model–JS	0.779	0.817	0.858

Equations

$$J=\sum _i^m\sum _{w_t\in V}\log P(w_t\mid C_{f_i}(w_t)),$$

$$P(w_t\mid C_{f_i}(w_t))=\frac{\exp ({\mathbf{w}}_t^{\top }\cdot vec(C_{f_i}(w_t)))}{{\sum }_{w\in D_{f_i}}\exp ({\mathbf{w}}_t^{\top }\cdot vec(C_{f_i}(w)))},$$

$$Di{s}^k(w_t^{f_i},w_t^{f_j})=\frac{\left|W_i^k\cup W_j^k\right|-\left|W_i^k\cap W_j^k\right|}{\left|W_i^k\cup W_j^k\right|}.$$

$$si{m}^k(w_t^{f_i},f_j)=\frac{{\sum }_{w\in S_{t,f_i,f_j}^k}f(w_t^{f_i})\cdot \cos (w_t^{f_i},w)\cdot f(w)}{\left|S_{t,f_i,f_j}^k\right|},$$

$$FieldDi{s}^k(f_i,f_j)=\frac{\exp (1-{\sum }_{w_t^{f_i}\in V_{f_i}}si{m}^k(w_t^{f_i},f_j))-1}{\lambda }.$$

Full text

I Introduction

The usage of language always varies among people with different backgrounds. When it comes to scientific literature, linguistic variation commonly exists across different scientific fields, among which we often see that scholars with varied backgrounds of knowledge use the same terms to express entirely different meanings. For this paper, we mainly consider different research fields within one subject (Computer Science) such as Natural Language Processing (NLP) and Computer Networks and Communication (Comm) instead of different subjects. This is because research fields within one subject usually have more shared terms whose semantic changes may lead to ambiguity [1]. Consider the term alignment, which often refers to the matching of signals in the field of computer communications, is however more typically related to translation of words or sentences in NLP research. Other representative examples include terms such as embedding, semantic, and grid, etc. Against this issue, quantified analysis of linguistic variation for scientific terms benefits with clearer understanding of concept expressions in different scientific fields, and reducing confusions for interdisciplinary communications. Moreover, such computational methods can also demonstrate the divergence of overall language trends among research fields.

While computational analysis methods of linguistic variation have attracted much attention recently, past work mainly focused on geographic, temporal, and social aspects of languages [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. Studies on these aspects mainly focus on differentiating word representations that change dramatically in the corresponding aspect for the entire vocabulary. However, existing work has not really tackled field variation of the language that modifies the meanings of scientific terms across these research fields, nor have they utilized these terms to reveal the general inter-field divergence of the language.

In this paper, we propose a computational method to analyze the linguistic variation in scientific research fields based on the quantification of semantic changes of crucial scientific terms. We extend the neural language model [12, 13] with a partially-localized mechanism to capture field-specific embeddings for frequent key terms, and preserve the global representations for other general words. Fig. 1 shows the the embedded word vectors for key terms in five domains after reducing the dimension of the embeddings, and we can see clear separation between domains. We also propose metrics to quantify the linguistic variation for scientific fields based on the semantic distances of key terms. The proposed approach hence helps keep track of the variation of term expressions, and provides evidences for speculations about the diversity of language usage across different research areas.

II Modeling

We begin with the formalization of the corpus $D$, which is a sequence of words. We use $F=\left\{f_{1},f_{2},...,f_{m}\right\}$ to denote the set of fields. $D$ is partitioned into $m$ disjoint field-specific corpora, i.e., $D=\bigcup_{f_{i}\in F}D_{f_{i}}$. We use $V=V_{G}\cup V_{F}$ to denote the vocabulary of words, for which $V_{G}=\left\{w_{t}\right\}$ is the global vocabulary, and $V_{F}=\bigcup_{f_{i}\in F}V_{f_{i}}$, which is disjoint with $V_{G}$, is the vocabulary of field-specific terms, where $V_{f_{i}}=\left\{w_{t}^{f_{i}\in F}\right\}$. In practice, $V_{F}$ can be predefined by selecting the most frequent terms from titles. Note that for every $f_{i},f_{j}\in F$, $w_{t}^{f_{i}}$ and $w_{t}^{f_{j}}$ represent one term $w_{t}$ that occurs in two field-specific corpora, and thus have two different embeddings. We establish two disjoint sets of vocabularies in order to localize embeddings for key terms and keep generic embeddings for other sentence elements. For a word $w_{t}\in D_{f_{i}}$, $C_{f_{i}}(w_{t})=\left\{w_{t-s},...,w_{t-1},w_{t+1},...,w_{t+s}\right\}$ is the context of $w_{t}$, where $s$ is half of the context size, and each word in $C_{f_{i}}(w_{t})$ belongs to either $V_{G}$ or $V_{f_{i}}$.

II-A Neural Language Model

We extend the CBOW model [12] to capture both global embeddings of words and field-specific embeddings of scientific terms. The training objective of the model is to learn word vector representations that maximize the log likelihood of the word and term given its context that is specified in different fields:

[TABLE]

for which the conditional likelihood of word $w_{t}$ over a field-specific context is defined as:

[TABLE]

where $\mathbf{w}_{t}$ is the embedding of word $w_{t}$, and context $C_{f_{i}}(w_{t})$ is represented as the mean of contained word embeddings, i.e., $vec(C_{f_{i}}(w_{t}))=\frac{1}{2s}\sum_{w_{i}\in C_{f_{i}(w_{t})}}\mathbf{w}_{i}$. Following [12], we adopt batched negative sampling to obtain a computationally efficient approximation of log likelihood.

II-B Quantification Approaches

In this section, we present several methods to quantify the linguistic variation for terms and research fields.

II-B1 Term Variation

We consider both Cosine Similarity: $\mathrm{cos}\left(w_{i},w_{j}\right)=\frac{\mathbf{w_{i}}\cdot\mathbf{w_{j}}}{\left\|\mathbf{w_{i}}\right\|_{2}\left\|\mathbf{w_{j}}\right\|_{2}}$ and Jaccard Similarity for quantitative analysis. To get the Jaccard Similarity, we find the most similar $k$ words or terms using cosine similarity for a term $w_{t}^{f}$ in $V_{F}$, based on which the variation between the term in two fields (i.e. $w_{t}^{f_{i}},w_{t}^{f_{j}}$) is calculated with Jaccard Distance. This similarity metric aims to capture the second order similarity for different terms. We consider the set of $k$ most similar words in order to capture the latent semantic of the term:

[TABLE]

$W^{k}_{i}$ thereof, indicates the set of $k$ most similar words considering its cosine similarities with $w_{i}$, i.e., $W^{k}_{i}=\left\{w_{t}\mid\mathrm{cos}(w_{t},w_{i})\in\mathrm{topk}(\bigcup_{w\in V}\left\{\mathrm{cos}(w_{i},w)\right\})\right\}$, for which $\mathrm{topk}(\bigcup_{w\in V}\left\{\mathrm{cos}(w_{i},w)\right\})$ denotes $k$ most similar words to $w_{i}$, and $J(W^{k}_{i},W^{k}_{j})=\frac{\left|W^{k}_{i}\cap W^{k}_{j}\right|}{\left|W^{k}_{i}\cup W^{k}_{j}\right|}$ is the Jaccard Index to measure the similarity of two sets. Intersection thereof is calculated by aggregating the cosine similarities of the words that appear in $W^{k}_{i}$ and $W^{k}_{j}$, and union is calculated by summing up the cosine similarities thereof.

In practice, a higher $k$ should cause $Dis^{k}$ to more comprehensively measure the variation of a term by considering the semantic distances of more related words in each field, which however requires more computational cost.

II-B2 Field Variation

We quantify the overall linguistic variation of research fields $f_{i},f_{j}$ based on the field-specific term vocabularies $V_{f_{i}},V_{f_{j}}$. For each term $w^{f_{i}}_{t}\in V_{f_{i}}$, we first find the set of $k$ most similar words $W_{t,f_{i}}^{k}$. Then the subset of $W_{t,f_{i}}^{k}$ that contains terms of the second field $f_{j}$ is obtained as $S_{t,f_{i},f_{j}}^{k}=\left\{w_{t}^{f_{j}}\mid w_{t}^{f_{j}}\in(W_{t,f_{i}}^{k}\bigcap V_{f_{j}})\right\}$. Based on that, we aggregate a term-to-field semantic similarity measure $sim^{k}(w_{t}^{f_{i}},f_{j})$, i.e.,

[TABLE]

where $f(w^{f_{i}}_{t})$ is the frequency of word $w^{f_{i}}_{t}$, i.e. number of occurrence of word $w^{f_{i}}_{t}$ in $D_{f_{i}}$ divided by the total number of words in $D_{f_{i}}$.

Then the semantic distance between two fields $f_{i},f_{j}$ is defined based on the aggregated term-to-field similarities for all words in $V_{f_{i}}$, for which we apply normalized exponential scaling [14] to signify the differences of measures for different fields. $\lambda=e-1$ thereof is a normalizing constant. Thus we have:

[TABLE]

III Evaluation

In this section, we evaluate the proposed approach for analyzing term and field-level linguistic variation based on a large collection of scientific literature. We collect human annotations on term variation and compare the results with separately-trained CBOW, GEODIST [3], and our model.

III-A Dataset and Model Configuration

We train our model on research papers in five fields of computer science, including NLP, Computer Vision (CV), Databases and Data Mining (DBDM), Computer Architecture (Arch), and Computer Networks and Communication (Comm). We select around ten mainstream conferences and journals for each field based on the Google Scholar Publication Ranking List 111https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=eng, and correspondingly collected the papers in these venues that are published in the past seven years. The corpora contain the plaintext contents of over 56K academic papers.

We populate $V_{F}$ using the article titles, where the stop words are removed and 200 most frequent terms are selected for each field. Since the title should reflect the main topic of each paper, this word collection naturally contains a set of very popular scientific terms. We set the dimensionality of embeddings to be 100, the size of the contexts $s$ to be 24, and the negative sample size to be 5.

III-B Variation of Terms

To evaluate the semantic change of a term $w_{t}$, we calculate the $Dis^{k}$ with the field-specific embeddings of this term, i.e. $\mathbf{w}^{f_{i}}_{t}$ and $\mathbf{w}^{f_{j}}_{t}$. Specifically, we analyze the $Dis^{k}$ of one term for in total ten pairs of fields, and for which average of their distances to quantify the overall variation of the term. We set $k$ to be 10,000 during the evaluation, which seeks to aggregate semantics from a large neighborhood of each term. Results reported in Table II show the ten terms with the overall most significant semantic changes, and between which fields such changes happen the most, while those in Table II show the ten terms with the least semantic change. The model detects that the meanings of terms like alignment and embedding should vary a lot across different fields, whereas those of terms like hardware and server are expected to be consistent. Table III presents the most similar words of some terms from Table II and Table II in each of the five field. It is noteworthy that, such terms with a high $Dis^{k}$ indeed have very different meanings and usages across different fields.

III-C Comparison on Human Annotated Data

To quantitatively evaluate the term variation found by our model, we randomly choose 50 terms and instruct a group of 30 Computer Science PhD students to annotate them as semantically varied (+1) or not (-1), across the five research fields. We make sure each single term is annotated by at least five PhD students, sum their annotations up, and divide the summed number by 5 to get the annotated term variation. We collect all annotation for 50 terms and compare with the corresponding variation returned from different methods as shown in Table IV. As a baseline, separately-trained CBOW model on each of the five domains is considered, and the variation is calculated using Jaccard Similarity mentioned before. We also ran the GEODIST model introduced by [3] on the corpus for comparison. Then we test the performances on our model using both Cosine Similarity and Jaccard Similarity as similarity measures. For metrics, we use Pearson Correlation ($\rho_{Pearson}$) [15] and Normalized Discounted Cumulative Gain (nDCG) at rank 10 and 50 [16]. From the results, we can see that our model performs better than baselines for all three metrics. For Pearson Correlation ($\rho_{Pearson}$), the improvement is more significant and the calculated variation has a linear correlation with the ground truth annotation. GEODIST fails to differentiate the semantic variation of terms across different groups of literature with selected terms, thus obtains a negative correlation and a low nDCG score. We hypothesis that it is because for the research field corpus, the language style of individual paper in the same field varies greatly and it is a more noisy corpus than the Google Book Ngrams corpus they used. Our model works well with the noisy data and also outperforms the separately-traind CBOW baseline. We also observe that using Jaccard similarity improves the results, showing that second order similarity metrics are arguably better than first order metrics.

III-D Variation of Fields

We now analyze the overall linguistic variation of different research fields based on $FieldDis^{k}$, for which we set $k$ as 10,000. The results are shown in the heatmap of Fig. 2, in which darker colors indicate larger overall semantic changes (i.e. higher $FieldDis^{k}$). The results indicate that the language usages between NLP and CV are considered as most similar. This is explainable because NLP and CV are both AI-related fields and share lots of techniques in research studies. For DBDM, corresponding linguistic variation from other fields is more significant. This can be explained by the fact that, on one hand, data mining tasks employ many shared statistical techniques in AI research as well, and a portion of modern database research in distributed scenarios also shares much common knowledge with computer and network systems. On the other hand, there is still much distinction from DBDM to the rest involved fields. Another observation is that the language usages are considered as most dissimilar from the fields of NLP and CV to the fields of Arch and Comm. This conclusion is congruent to the evaluation results for terms in Table II, where distances for these fields are the highest for terms with high semantic changes. This is because AI research topics are often considered distant from computer system-related research fields like Arch and Comm, which explains why the $FieldDis^{k}$ measures are significant from any of the former to that of the later. Besides, we discover that the semantic change between Arch and Comm is also relatively significant.

IV Related Work

In this section, we discuss related work on computational methods for linguistic variation. Recent work has paid much attention to using neural language models for corresponding analysis, which are respectively based on two lines of word representation mechanisms.

Diffusion-based. The diffusion-based mechanism categorizes the corpora by different context scenarios, and obtains differentiated representations for words in each context scenario. Corresponding methods have been used to analyze the semantics of words that vary according to different time periods or social groups [5, 4, 10, 2]. Our work is close to these methods, but instead of differentiating all words, we employs a partially-localized mechanism by localizing embeddings for a portion of important terms, while keep the universal representations for other generic words. This is due to that our task naturally focuses on measuring the change of frequent scientific terms that contribute mostly to the linguistic variation among scientific research fields.

Bias-based. Other work adopts the bias-based representation mechanism, which uniformly represents all words, and overlays a scenario-specific bias vector to words that appear in the corresponding context scenario. Exemplarily, [3] captures the biases between UK and US English based on corresponding literatures, and [6] induces such biases on social media corpora that are tagged with more fine-grained geo-locations. Similarly, a user-specific bias is utilized by [9]. While each context scenario often applies the same bias to its words, we prefer the diffusion-based representations due to that our task requires the semantic changes to be captured distinctively for the terms in the same field.

Besides, we also provide quantitaive evaluation for the task of analyzing scientific language variation for different domains. Previous work has mainly showed qualitative analysis on language variation geographically or temporally. We argue that by comparing with human annotations, our model can capture the semantic change for cross-domain data. Moreover, we leverage the word-level quantification to measure the overall semantic differences of research fields.

V Conclusion

In this paper we introduced a computational model for analyzing scientific linguistic variation by research fields. The model was trained on a large collection of literature from five computer science research fields to obtain partially-localized representations for terms. A series of metrics were provided to quantify the semantic change on both term and field level. We evaluate the term variation found by our model by comparing with human annotated data and sevearl baselines and show that our model captures the term variation most accurately. We believe that with automatically detected term variation, confusion during interdisciplinary communications is reduced, and the goal of better cross-domain data collaboration is achieved.