Semantic Matching of Documents from Heterogeneous Collections: A Simple and Transparent Method for Practical Applications

TL;DR

This paper introduces a simple, unsupervised, and transparent method for pairwise document matching across heterogeneous collections, outperforming more complex systems in a binary classification task.

Contribution

The authors propose a novel, straightforward approach that uses standard resources and explicit similarity computations, enhancing transparency and efficiency in document matching.

Findings

Outperforms complex baseline systems in Concept-Project matching

Uses only standard, pre-computable word similarities

Provides explicit, interpretable similarity scores

Abstract

We present a very simple, unsupervised method for the pairwise matching of documents from heterogeneous collections. We demonstrate our method with the Concept-Project matching task, which is a binary classification task involving pairs of documents from heterogeneous collections. Although our method only employs standard resources without any domain- or task-specific modifications, it clearly outperforms the more complex system of the original authors. In addition, our method is transparent, because it provides explicit information about how a similarity score was computed, and efficient, because it is based on the aggregation of (pre-computable) word-level similarities.

Tables

Table 1. Table 1: Tuning Data Results AVG_COS_SIM . Top F per Concept Input Type in Bold .

Concept Input $\to$			Label				Description				Both
Embeddings	TF	IDF	T	P	R	F	T	P	R	F	T	P	R	F
GloVe	-	-	$.635$	$.750$	$.818$	$.783$	$.720$	$.754$	$.891$	$.817$	$.735$	$.765$	$.945$	$.846$
	+	-	$.640$	$.891$	$.745$	$.812$	$.700$	$.831$	$.891$	$.860$	$.690$	$.813$	$.945$	$.874$
	-	+	$.600$	$.738$	$.873$	$.800$	$.670$	$.746$	$.909$	$.820$	$.755$	$.865$	$.818$	$.841$
	+	+	$.605$	$.904$	$.855$	$.879$	$.665$	$.857$	$.873$	$.865$	$.715$	$.923$	$.873$	$.897$
Google	-	-	$.440$	$.813$	$.945$	$.874$	$.515$	$.701$	$.982$	$.818$	$.635$	$.920$	$.836$	$.876$
	+	-	$.445$	$.943$	$.909$	.926	$.540$	$.873$	$.873$	$.873$	$.565$	$.927$	$.927$	$.927$
	-	+	$.435$	$.839$	$.945$	$.889$	$.520$	$.732$	$.945$	$.825$	$.590$	$.877$	$.909$	$.893$
	+	+	$.430$	$.943$	$.909$	.926	$.530$	$.889$	$.873$	.881	$.545$	$.945$	$.945$	.945
fastText	-	-	$.440$	$.781$	$.909$	$.840$	$.555$	$.708$	$.927$	$.803$	$.615$	$.778$	$.891$	$.831$
	+	-	$.435$	$.850$	$.927$	$.887$	$.520$	$.781$	$.909$	$.840$	$.530$	$.803$	$.964$	$.876$
	-	+	$.435$	$.850$	$.927$	$.887$	$.525$	$.722$	$.945$	$.819$	$.600$	$.820$	$.909$	$.862$
	+	+	$.420$	$.895$	$.927$	$.911$	$.505$	$.803$	$.891$	$.845$	$.520$	$.833$	$.909$	$.870$

Table 2. Table 2: Tuning Data Results TOP_n_COS_SIM_AVG . Top F per Concept Input Type in Bold .

Concept Input $\to$			Label				Description				Both
Embeddings	TF	IDF	T/n	P	R	F	T/n	P	R	F	T/n	P	R	F
GloVe	+	-	$.365/6$	$.797$	$.927$	$.857$	$.690/14$	$.915$	$.782$	$.843$	$.675/16$	$.836$	$.927$	$.879$
	-	+	$.300/30$	$.929$	$.236$	$.377$	$.300/30$	$.806$	$.455$	$.581$	$.300/30$	$.778$	$.636$	$.700$
	+	+	$.330/6$	$.879$	$.927$	$.903$	$.345/6$	$.881$	$.945$	.912	$.345/6$	$.895$	$.927$	$.911$
Google	+	-	$.345/22$	$.981$	$.927$	.953	$.480/16$	$.895$	$.927$	$.911$	$.520/16$	$.912$	$.945$	$.929$
	-	+	$.300/30$	$1.00$	$.345$	$.514$	$.300/8$	$1.00$	$.345$	$.514$	$.300/30$	$1.00$	$.600$	$.750$
	+	+	$.300/10$	$1.00$	$.509$	$.675$	$.300/14$	$.972$	$.636$	$.769$	$.350/22$	$1.00$	$.836$	$.911$
fastText	+	-	$.415/22$	$.980$	$.873$	$.923$	$.525/14$	$.887$	$.855$	$.870$	$.535/20$	$.869$	$.964$	$.914$
	-	+	$.350/24$	$1.00$	$.309$	$.472$	$.300/30$	$1.00$	$.382$	$.553$	$.300/28$	$1.00$	$.673$	$.804$
	+	+	$.300/20$	$1.00$	$.800$	$.889$	$.300/10$	$.953$	$.745$	$.837$	$.310/14$	$.963$	$.945$	.954

Table 3. Table 3: Test Data Results

						P		R		F
Gong et al. (2018)	topic_science					$.758$	$\pm .012$	$.885$	$\pm .071$	$.818$	$\pm .028$
	topic_wiki					$.750$	$\pm .009$	$.842$	$\pm .010$	$.791$	$\pm .007$
Method	Embeddings	Settings		T/n	Conc. Input
AVG_COS_SIM	Google	+TF	+IDF	$.515$	Label	$.939$	$\pm .043$	$.839$	$\pm .067$	$.884$	$\pm .038$
	Google	+TF	+IDF	$.520$	Description	$.870$	$\pm .068$	$.834$	$\pm .048$	$.849$	$\pm .038$
	Google	+TF	+IDF	$.545$	Both	$.915$	$\pm .040$	$.938$	$\pm .047$	.926	$\pm .038$
TOP_n_COS_SIM_AVG	Google	+TF	-IDF	$.345/22$	Label	$.854$	$\pm .077$	$.861$	$\pm .044$	$.856$	$\pm .054$
	GloVe	+TF	+IDF	$.345/6$	Description	$.799$	$\pm .063$	$.766$	$\pm .094$	$.780$	$\pm .068$
	fastText	+TF	+IDF	$.310/14$	Both	$.850$	$\pm .059$	$.918$	$\pm .049$	.881	$\pm .037$

Table 4. Table 4: TOP_n_COS_SIM_AVG Detail Results of Best-performing fastText Model on Both .

TP ($\text{.447}>.310$)			FP ($\text{.367}>.310$)			TN ()			FN ()
Concept	Project	Sim	Concept	Project	Sim	Concept	Project	Sim	Concept	Project	Sim
Word	Word		Word	Word		Word	Word		Word	Word
cells	enzymes	$.438$	co-evolution	dynamic	$.299$	energy	allergy	$.147$	area	water	$.277$
cells	genes	$.427$	continual	dynamic	$.296$	energy	juice	$.296$	climate	water	$.269$
molecules	DNA	$.394$	delicate	detail	$.306$	energy	leavening	$.186$	earth	copper	$.254$
molecules	enzyme	$.445$	delicate	dynamic	$.326$	energy	substitutes	$.177$	earth	metal	$.277$
molecules	enzymes	$.533$	delicate	texture	$.379$	surface	average	$.212$	earth	metals	$.349$
molecules	gene	$.369$	dynamic	dynamic	$1.00$	surface	baking	$.216$	earth	water	$.326$
molecules	genes	$.471$	dynamic	image	$.259$	surface	egg	$.178$	extent	concentration	$.266$
multiple	different	$.550$	dynamic	range	$.377$	surface	leavening	$.158$	range	concentration	$.255$
organisms	enzyme	$.385$	dynamic	texture	$.310$	surface	thickening	$.246$	range	ppm	$.237$
organisms	enzymes	$.512$	surface	level	$.323$	transfer	baking	$.174$	systems	metals	$.243$
organisms	genes	$.495$	surface	texture	$.383$	transfer	substitute	$.192$	systems	solution	$.275$
organs	enzymes	$.372$	surface	tiles	$.321$	transfer	substitutes	$.157$	typical	heavy	$.299$
tissues	enzymes	$.448$	systems	dynamic	$.272$	warms	baking	$.176$	weather	heavy	$.248$
tissues	genes	$.414$	systems	levels	$.286$	warms	thickening	$.214$	weather	water	$.308$
	Avg. Sim	.447		Avg. Sim	.367		Avg. Sim	.195		Avg. Sim	.278

Full text

1 Introduction

We present a simple and efficient unsupervised method for pairwise matching of documents from heterogeneous collections. Following Gong et al. (2018), we consider two document collections heterogeneous if their documents differ systematically with respect to vocabulary and / or level of abstraction. With these defining differences, there often also comes a difference in length, which, however, by itself does not make document collections heterogeneous. Examples include collections in which expert answers are mapped to non-expert questions (e.g. InsuranceQA by Feng et al. (2015)), but also so-called community QA collections (Blooma and Kurian (2011)), where the lexical mismatch between Q and A documents is often less pronounced than the length difference.

Like many other approaches, the proposed method is based on word embeddings as universal meaning representations, and on vector cosine as the similarity metric. However, instead of computing pairs of document representations and measuring their similarity, our method assesses the document-pair similarity on the basis of selected pairwise word similarities. This has the following advantages, which make our method a viable candidate for practical, real-world applications: efficiency, because pairwise word similarities can be efficiently (pre-)computed and cached, and transparency, because the selected words from each document are available as evidence for what the similarity computation was based on.

We demonstrate our method with the Concept-Project matching task (Gong et al. (2018)), which is described in the next section.

2 Task, Data Set, and Original Approach

The Concept-Project matching task is a binary classification task where each instance is a pair of heterogeneous documents: one concept, which is a short science curriculum item from NGSS111https://www.nextgenscience.org, and one project, which is a much longer science project description for school children from ScienceBuddies222https://www.sciencebuddies.org. The publicly available data set333https://github.com/HongyuGong/Document-Similarity-via-Hidden-Topics contains $510$ labelled pairs444Of the original $537$ labelled pairs, $27$ were duplicates, which we removed. involving $C=75$ unique concepts and $P=230$ unique projects. A pair is annotated as $1$ if the project matches the concept ($57\%$), and as [math] otherwise ($43\%$). The annotation was done by undergrad engineering students. Gong et al. (2018) do not provide any specification, or annotation guidelines, of the semantics of the ’matches’ relation to be annotated. Instead, they create gold standard annotations based on a majority vote of three manual annotations. Figure 1 provides an example of a matching C-P pair. The concept labels can be very specific, potentially introducing vocabulary that is not present in the actual concept descriptions. The extent to which this information is used by Gong et al. (2018) is not entirely clear, so we experiment with several setups (cf. Section 4).

2.1 Gong

et al. (2018)’s Approach

The approach by Gong et al. (2018) is based on the idea that the longer document in the pair is reduced to a set of topics which capture the essence of the document in a way that eliminates the effect of a potential length difference. In order to overcome the vocabulary mismatch, these topics are not based on words and their distributions (as in LSI (Deerwester et al. (1990)) or LDA (Blei et al. (2003))), but on word embedding vectors. Then, basically, matching is done by measuring the cosine similarity between the topic vectors and the short document words. Gong et al. (2018) motivate their approach mainly with the length mismatch argument, which they claim makes approaches relying on document representations (incl. vector averaging) unsuitable. Accordingly, they use Doc2Vec (Le and Mikolov (2014)) as one of their baselines, and show that its performance is inferior to their method. They do not, however, provide a much simpler averaging-based baseline. As a second baseline, they use Word Mover’s Distance (Kusner et al. (2015)), which is based on word-level distances, rather than distance of global document representations, but which also fails to be competitive with their topic-based method. Gong et al. (2018) use two different sets of word embeddings: One (topic_wiki) was trained on a full English Wikipedia dump, the other (wiki_science) on a smaller subset of the former dump which only contained science articles.

3 Our Method

We develop our method as a simple alternative to that of Gong et al. (2018). We aim at comparable or better classification performance, but with a simpler model. Also, we design the method in such a way that it provides human-interpretable results in an efficient way. One common way to compute the similarity of two documents (i.e. word sequences) $c$ and $p$ is to average over the word embeddings for each sequence first, and to compute the cosine similarity between the two averages afterwards. In the first step, weighting can be applied by multiplying a vector with the TF, IDF, or TFIDF score of its pertaining word. We implement this standard measure (AVG_COS_SIM) as a baseline for both our method and for the method by Gong et al. (2018). It yields a single scalar similarity score. The core idea of our alternative method is to turn the above process upside down, by computing the cosine similarity of selected pairs of words from $c$ and $p$ first, and to average over the similarity scores afterwards (cf. also Section 6). More precisely, we implement a measure TOP_n_COS_SIM_AVG as the average of the $n$ highest pairwise cosine similarities of the $n$ top-ranking words in $c$ and $p$. Ranking, again, is done by TF, IDF, and TFIDF. For each ranking, we take the top-ranking $n$ words from $c$ and $p$, compute $n\times n$ similarities, rank by decreasing similarity, and average over the top $n$ similarities. This measure yields both a scalar similarity score and a list of $<c_{x},p_{y},sim>$ tuples, which represent the qualitative aspects of $c$ and $p$ on which the similarity score is based.

4 Experiments

Setup

All experiments are based on off-the-shelf word-level resources: We employ WOMBAT (Müller and Strube (2018)) for easy access to the 840B GloVe (Pennington et al. (2014)) and the GoogleNews555https://code.google.com/archive/p/word2vec/ Word2Vec (Mikolov et al. (2013)) embeddings. These embedding resources, while slightly outdated, are still widely used. However, they cannot handle out-of-vocabulary tokens due to their fixed, word-level lexicon. Therefore, we also use a pretrained English fastText model666https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz (Bojanowski et al. (2017); Grave et al. (2018)), which also includes subword information. IDF weights for approx. $12$ mio. different words were obtained from the English Wikipedia dump provided by the Polyglot project (Al-Rfou et al. (2013)). All resources are case-sensitive, i.e. they might contain different entries for words that only differ in case (cf. Section 5).

We run experiments in different setups, varying both the input representation (GloVe vs. Google vs. fastText embeddings, $\pm$ TF-weighting, and $\pm$ IDF-weighting) for concepts and projects, and the extent to which concept descriptions are used: For the latter, Label means only the concept label (first and second row in the example), Description means only the textual description of the concept, and Both means the concatenation of Label and Description. For the projects, we always use both label and description. For the project descriptions, we extract only the last column of the original file (CONTENT), and remove user comments and some boiler-plate. Each instance in the resulting data set is a tuple of $<c,p,label>$, where $c$ and $p$ are bags of words, with case preserved and function words777We use the list provided by Gong et al. (2018), with an additional entry for cannot. removed, and $label$ is either [math] or $1$.

Parameter Tuning

Our method is unsupervised, but we need to define a threshold parameter which controls the minimum similarity that a concept and a project description should have in order to be considered a match. Also, the TOP_n_COS_SIM_AVG measure has a parameter $n$ which controls how many ranked words are used from $c$ and $p$, and how many similarity scores are averaged to create the final score. Parameter tuning experiments were performed on a random subset of $20\%$ of our data set ($54$% positive). Note that Gong et al. (2018) used only $10\%$ of their $537$ instances data set as tuning data. The tuning data results of the best-performing parameter values for each setup can be found in Tables 1 and 2. The top F scores per type of concept input (Label, Description, Both) are given in bold. For AVG_COS_SIM and TOP_n_COS_SIM_AVG, we determined the threshold values (T) on the tuning data by doing a simple $.005$ step search over the range from $0.3$ to $1.0$. For TOP_n_COS_SIM_AVG, we additionally varied the value of $n$ in steps of $2$ from $2$ to $30$.

Results

The top tuning data scores for AVG_COS_SIM (Table 1) show that the Google embeddings with TF*IDF weighting yield the top F score for all three concept input types ($.881$ - $.945$). Somewhat expectedly, the best overall F score ($.945$) is produced in the setting Both, which provides the most information. Actually, this is true for all four weighting schemes for both GloVe and Google, while fastText consistently yields its top F scores ($.840$ - $.911$) in the Label setting, which provides the least information. Generally, the level of performance of the simple baseline measure AVG_COS_SIM on this data set is rather striking.

For TOP_n_COS_SIM_AVG, the tuning data results (Table 2) are somewhat more varied: First, there is no single best performing set of embeddings: Google yields the best F score for the Label setting ($.953$), while GloVe (though only barely) leads in the Description setting ($.912$). This time, it is fastText which produces the best F score in the Both setting, which is also the best overall tuning data F score for TOP_n_COS_SIM_AVG ($.954$). While the difference to the Google result for Label is only minimal, it is striking that the best overall score is again produced using the ’richest’ setting, i.e. the one involving both TF and IDF weighting and the most informative input.

We then selected the best performing parameter settings for every concept input and ran experiments on the held-out test data. Since the original data split used by Gong et al. (2018) is unknown, we cannot exactly replicate their settings, but we also perform ten runs using randomly selected $10\%$ of our $408$ instances test data set, and report average P, R, F, and standard deviation. The results can be found in Table 3. For comparison, the two top rows provide the best results of Gong et al. (2018).

The first interesting finding is that the AVG_COS_SIM measure again performs very well: In all three settings, it beats both the system based on general-purpose embeddings (topic_wiki) and the one that is adapted to the science domain (topic_science), with again the Both setting yielding the best overall result ($.926$). Note that our Both setting is probably the one most similar to the concept input used by Gong et al. (2018). This result corroborates our findings on the tuning data, and clearly contradicts the (implicit) claim made by Gong et al. (2018) regarding the infeasibility of document-level matching for documents of different lengths. The second, more important finding is that our proposed TOP_n_COS_SIM_AVG measure is also very competitive, as it also outperforms both systems by Gong et al. (2018) in two out of three settings. It only fails in the setting using only the Description input.888Remember that this setup was only minimally superior ($.001$ F score) to the next best one on the tuning data. This is the more important as we exclusively employ off-the-shelf, general-purpose embeddings, while Gong et al. (2018) reach their best results with a much more sophisticated system and with embeddings that were custom-trained for the science domain. Thus, while the performance of our proposed TOP_n_COS_SIM_AVG method is superior to the approach by Gong et al. (2018), it is itself outperformed by the ’baseline’ AVG_COS_SIM method with appropriate weighting. However, apart from raw classification performance, our method also aims at providing human-interpretable information on how a classification was done. In the next section, we perform a detail analysis on a selected setup.

5 Detail Analysis

The similarity-labelled word pairs from concept and project description which are selected during classification with the TOP_n_COS_ SIM_AVG measure provide a way to qualitatively evaluate the basis on which each similarity score was computed. We see this as an advantage over average-based comparison (like AVG_COS_SIM), since it provides a means to check the plausibility of the decision. Here, we are mainly interested in the overall best result, so we perform a detail analysis on the best-performing Both setting only (fastText, TF*IDF weighting, $T=.310$, $n=14$). Since the Concept-Project matching task is a binary classification task, its performance can be qualitatively analysed by providing examples for instances that were classified correctly (True Positive (TP) and True Negative (TN)) or incorrectly (False Positive (FP) and False Negative (FN)).

Table 4 shows the concept and project words from selected instances (one TP, FP, TN, and FN case each) of the tuning data set. Concept and project words are ordered alphabetically, with concept words appearing more than once being grouped together. According to the selected setting, the number of word pairs is $n=14$. The bottom line in each column provides the average similarity score as computed by the TOP_n_COS_SIM_AVG measure. This value is compared against the threshold $T=.310$. The similarity is higher than $T$ in the TP and FP cases, and lower otherwise. Without going into too much detail, it can be seen that the selected words provide a reasonable idea of the gist of the two documents. Another observation relates to the effect of using unstemmed, case-sensitive documents as input: the top-ranking words often contain inflectional variants (e.g. enzyme and enzymes, level and levels in the example), and words differing in case only can also be found. Currently, these are treated as distinct (though semantically similar) words, mainly out of compatibility with the pretrained GloVe and Google embeddings. However, since our method puts a lot of emphasis on individual words, in particular those coming from the shorter of the two documents (the concept), results might be improved by somehow merging these words (and their respective embedding vectors) (see Section 7).

6 Related Work

While in this paper we apply our method to the Concept-Project matching task only, the underlying task of matching text sequences to each other is much more general. Many existing approaches follow the so-called compare-aggregate framework (Wang and Jiang (2017)). As the name suggests, these approaches collect the results of element-wise matchings (comparisons) first, and create the final result by aggregating these results later. Our method can be seen as a variant of compare-aggregate which is characterized by extremely simple methods for comparison (cosine vector similarity) and aggregation (averaging). Other approaches, like He and Lin (2016) and Wang and Jiang (2017), employ much more elaborated supervised neural networks methods. Also, on a simpler level, the idea of averaging similarity scores (rather than scoring averaged representations) is not new: Camacho-Collados and Navigli (2016) use the average of pairwise word similarities to compute their compactness score.

7 Conclusion and Future Work

We presented a simple method for semantic matching of documents from heterogeneous collections as a solution to the Concept-Project matching task by Gong et al. (2018). Although much simpler, our method clearly outperformed the original system in most input settings. Another result is that, contrary to the claim made by Gong et al. (2018), the standard averaging approach does indeed work very well even for heterogeneous document collections, if appropriate weighting is applied. Due to its simplicity, we believe that our method can also be applied to other text matching tasks, including more ’standard’ ones which do not necessarily involve heterogeneous document collections. This seems desirable because our method offers additional transparency by providing not only a similarity score, but also the subset of words on which the similarity score is based. Future work includes detailed error analysis, and exploration of methods to combine complementary information about (grammatically or orthographically) related words from word embedding resources. Also, we are currently experimenting with a pretrained ELMo (Peters et al. (2018)) model as another word embedding resource. ELMo takes word embeddings a step further by dynamically creating contextualized vectors from input word sequences (normally sentences). Our initial experiments have been promising, but since ELMo tends to yield different, context-dependent vectors for the same word in the same document, ways have still to be found to combine them into single, document-wide vectors, without (fully) sacrificing their context-awareness.

The code used in this paper is available at https://github.com/nlpAThits/TopNCosSimAvg.

Acknowledgements The research described in this paper was funded by the Klaus Tschira Foundation. We thank the anonymous reviewers for their useful comments and suggestions.