TL;DR
This paper introduces a new method for adjusting term weights in document representations by redistributing weights based on contextual similarity, improving semantic accuracy in unsupervised text classification.
Contribution
The paper proposes a novel contextually propagated term weighting model that enhances bag-of-words representations by sharing semantic information among similar words.
Findings
Outperforms 8 state-of-the-art baselines in F1 scores
Effective across datasets of varying difficulty
Improves semantic coherence in document representations
Abstract
Word embeddings predict a word from its neighbours by learning small, dense embedding vectors. In practice, this prediction corresponds to a semantic score given to the predicted word (or term weight). We present a novel model that, given a target word, redistributes part of that word's weight (that has been computed with word embeddings) across words occurring in similar contexts as the target word. Thus, our model aims to simulate how semantic meaning is shared by words occurring in similar contexts, which is incorporated into bag-of-words document representations. Experimental evaluation in an unsupervised setting against 8 state of the art baselines shows that our model yields the best micro and macro F1 scores across datasets of increasing difficulty.
| Dataset | #docs | unique words | #avg. unique words std. | #classes |
|---|---|---|---|---|
| bbcsport | 737 | 13106 | 116.455.9 | 5 |
| 3424 | 8405 | 8.93.3 | 3 | |
| classic | 7095 | 23624 | 39.328.1 | 4 |
| amazon | 8000 | 39852 | 44.4746.9 | 4 |
| reuters | 7674 | 23109 | 38.936.9 | 8 |
| 20news | 18846 | 30465 | 84.196.9 | 20 |
| wiki | 19981 | 46610 | 474.9411.7 | 25 |
| F | BM25 | BOW | TF-IDF | LDA | LSI | WE-AVG | WMD | SIF | CPTW | CPTW |
|
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
micro macro |
|
| bbcsport | .986 .987 | |||||||||
| .732 .550 | ||||||||||
| classic | .963 .964 | |||||||||
| amazon | .918 .918 | |||||||||
| reuters | .961 .910 | |||||||||
| 20news | .703 .699 | |||||||||
| wiki | ||||||||||
| allWMD | .740 .663 | .780 .705 | .784 .712 | .700 .613 | .770 .697 | .783 .719 | .785 .716 | .797 .732 | .806 .740 | |
| allWMD | .864 .805 | .883 .829 | .887 .838 | .805 .731 | .881 .821 | .887 .837 | .903 .841 | .889 .833 | .902 .849 | .907 .862 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Contextually Propagated Term Weights for Document Representation
Casper Hansen
University of Copenhagen
,
Christian Hansen
University of Copenhagen
,
Stephen Alstrup
University of Copenhagen
,
Jakob Grue Simonsen
University of Copenhagen
and
Christina Lioma
University of Copenhagen
(2019)
Abstract.
Word embeddings predict a word from its neighbours by learning small, dense embedding vectors. In practice, this prediction corresponds to a semantic score given to the predicted word (or term weight). We present a novel model that, given a target word, redistributes part of that word’s weight (that has been computed with word embeddings) across words occurring in similar contexts as the target word. Thus, our model aims to simulate how semantic meaning is shared by words occurring in similar contexts, which is incorporated into bag-of-words document representations. Experimental evaluation in an unsupervised setting against 8 state of the art baselines shows that our model yields the best micro and macro F1 scores across datasets of increasing difficulty.
Word embeddings, Contextual semantics, Document representation
††journalyear: 2019††copyright: acmcopyright††conference: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval; July 21–25, 2019; Paris, France††booktitle: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19), July 21–25, 2019, Paris, France††price: 15.00††doi: 10.1145/3331184.3331307††isbn: 978-1-4503-6172-9/19/07
1. Introduction
Word embeddings represent words as elements in a learned vector space by mapping semantically similar words to nearby points (otherwise put, by embedding them nearby each other). By doing so, word embeddings adopt the Distributional Hypothesis, which posits that words appearing in the same contexts share semantic meaning (Lioma et al., 2015). An efficient and popular implementation for learning word embeddings is word2vec (Mikolov et al., 2013b) and the skip-gram model. Given a target word, skip-gram is trained to predict the words in the context of that target word. This prediction practically corresponds to a score (or weight) given to the predicted word, which can be applied to a range of tasks.
We present a novel model111The code is available at: https://github.com/casperhansen/CPTW that, given a target word, redistributes part of that word’s weight (that has been computed with word embeddings) back to words occurring in similar contexts as the target word. By doing so, our model aims to simulate how semantic meaning is shared by words occurring in the same context, by sharing (or propagating) the semantic scores computed for these words within the neighbourhood of contextually similar words. This propagation is incorporated into bag-of-words document representations. We experimentally evaluate our model in the task of unsupervised text clustering against 3 established and 5 state of the art baselines. Our model yields the best micro and macro scores on average across all datasets. In addition, our model is efficient and robust across datasets of different inter versus intra class ratio (i.e. across datasets of increasing difficulty).
2. Related Work
A simple, yet often effective practice of outputting document scores from word embeddings is by the average of the embedding of each word in a text (Lev et al., 2015). A disadvantage of this approach is that all words are weighted equally, failing to distinguish between more and less discriminative words. Arora et al. (Arora et al., 2017) proposed an unsupervised word embedding weighting scheme using word occurrence probabilities from a large corpus. Each embedded word is weighted by: with being the probability of word occurring, and being a smoothing parameter. From each averaged vector the first singular vector is subtracted, which corresponds to removing common words (e.g. just, when, and even) (Arora et al., 2017). The method of Arora et al. was defined and evaluated for sentences. Kusner et al. (Kusner et al., 2015) recently presented the unsupervised word mover’s distance (WMD), which approximates the semantic distance between two documents by computing the earth mover’s distance between the embedded words in each document. While WMD experimentally outperformed several state of the art methods in the task of text clustering, its main disadvantage is that it is computationally very costly and cannot be readily used on medium to large scale datasets.
3. Contextually Propagated Term Weights
Given a word embedding (Mikolov et al., 2013a), we define as the embedded neighbourhood of term in document , the set of all similar terms having cosine similarity to at least in the embedding space (i.e. the most similar terms thresholded by a minimum cosine similarity of ), where is a tunable threshold value (discussed in Section 3.3)222Note that embedded neighbourhood is not the same as embedding space: the former is derived from the similarities measured in the latter.. We use this embedded neighbourhood to define two contextual document representations, presented next.
3.1. CPTW
Let be the set of words contained in the embedded neighbourhood of word (note that this includes itself). Then, for each , we define:
[TABLE]
where is our contextually propagated term weight of , is the frequency of word in document , and is the cosine similarity between the word embeddings for word and . Eq. 1 computes the term weight of and of each word in ’s embedded neighbourhood. Then, based on the above term weights, we compute the contextually propagated term weights (CPTW) of document , denoted , as:
[TABLE]
where is the total number of unique words in the collection (such that the representation resides in ), is the vector with at index and zero everywhere else, is the term weight of computed using Eq. 1, and is a normalization constant, computed as: . In Eq. 2, ensures that all words have the same total weight independent of the number of words they are similar to. This has the benefit that a word with a larger embedded neighbourhood (larger ) is not weighted higher than a word with a smaller embedded neighbourhood (smaller ). When the number of unique words is considered fixed then a -tresholded word-to-word similarity matrix can be computed offline, such that the similarity propagation of CPTW can be trivially done efficiently using sparse vector-matrix multiplications on a traditional bag-of-words representation. Fig. 1 shows an example of CPTW in practice.
3.2.
Eq. 1 approximates the weight of term according to (i) the frequency of its occurrence in (), and (ii) how similar is to (). The frequency of in can be artificially inflated for terms that occur very often in the collection, not just in (just like TF in traditional bag of words computations when not combined with IDF). To counter this effect, we introduce the following variation of that includes an IDF-like component, thus ensuring that high within-document term frequency and term discriminativeness in the collection are considered when computing term weights:
[TABLE]
where is computed as in Eq. 1, is the number of documents in the collection that contain word , is the total number of documents in the collection, and the rest of the notation is the same as above. Note that in this case is placed inside the because we propagate the IDF component and still need the normalization for the same reason as above. We define the of document based on the above term weights as:
[TABLE]
The purpose of the IDF component is the same as in traditional TF-IDF; however, in our approach it is computed by propagating the IDF values for each word in the embedded neighbourhood as done similarly for term frequency in Eq. 2. This means that the IDF score for each word is based on a weighted sum of all the IDF scores of the words in its embedded neighbourhood, thus taking into account the differences in IDF scores between the words in the embedded neighbourhood.
3.3. Threshold parameter
The optimal value of depends on both (i) the quality of the word embedding (to avoid dissimilar words from being represented as similar), and (ii) the similarity of the texts in the collection. If the collection consists of texts with vastly different topics, then a low is preferred because there is little overlap of word semantics between texts with different topics. However, if the topics are similar, then should be higher to avoid including semi-related words that are shared across topics. If, for the sake of explanation, we assume that no thresholding is done, then each word contains all words in its embedded neighbourhood, and thus the value of each word in the bag-of-words vector would be the sum of cosine similarities to all other words. As the cosine similarity is a dot product of unit length vectors, the distributive property of the dot product entails that the sum of cosine similarities would be, in effect, the dot product of the embedded word vector and the sum of all other embedded word vectors. In order to not consider this sum of all other embedded word vectors, we choose to define a smaller neighbourhood such that the sum only consists of words that are deemed similar enough to not introduce noise (i.e. minuscule or negative dot products).
4. Experimental Evaluation
We experimentally evaluate our contextually propagated term weights (CPTW) and CPTW against strong baselines for related document representation and document distance methods that use few to zero parameters, applied to text clustering. For all methods we apply k nearest neighbour (kNN) classification for evaluation purposes, in order to purely focus on the representation that each method can generate to discriminative between texts.
4.1. Data
We use 7 openly available datasets commonly used in related work (Kusner et al., 2015; Kenter and De Rijke, 2015; J et al., 2016; Zhang et al., 2016). As seen in Table 1, the datasets are largely varied with respect to their vocabulary size, number of unique words in each document, and number of classes.
4.2. Baselines and Tuning
We compare our CPTW and CPTW models against bag-of-words with frequency weighting (BOW) (Salton and Buckley, 1988), TF-IDF (Salton and Buckley, 1988), BM25 (Robertson et al., 1995), LSI (Deerwester et al., 1990), LDA (Blei et al., 2003), Word Embedding Averaging (WE-AVG) (Lev et al., 2015), Smooth inverse frequency weighting (SIF) (Arora et al., 2017), and Word Mover’s Distance (WMD) (Kusner et al., 2015). To tune and evaluate all models we use 5-fold cross validation, where each fold acts as testing data once, while the rest is used for training and validation. In each iteration, we use 70% of the 4 folds for training and the remaining 30% for validation. We tune the parameters of all models using a grid search of the parameter space and repeat this 3 times in each iteration (the best parameters across the 3 iterations are chosen for the test fold). For reuters, 20news, and wiki we use the existing splits.
We use the micro and macro versions of for evaluation, and for validation we use the micro (corresponding to traditional accuracy) for choosing the best parameters. In all methods we search kNN’s as done in a similar setup by Kusner et al. (Kusner et al., 2015), and also tune the following: For LSI and LDA the number of topics is searched in . For BM25 and . For SIF . For CPTW and CPTW . We use the best parameters found on the validation set when evaluating on the test set.
We use word embedding pretrained by the word2vec skip-gram model on Google News data333https://code.google.com/archive/p/word2vec/. For reproducibility purposes we use the gensim and scikit-learn python libraries444https://radimrehurek.com/gensim/ and https://scikit-learn.org/stable/ for all compared methods, and use the implementation of SIF provided by the authors555https://github.com/PrincetonML/SIF. We preprocess the datasets to remove non-alphanumeric characters, to tokenize text into lower cased word-tokens, and to remove stop words (based on the list by Salton (Salton, 1971)) as per (Kusner et al., 2015).
4.3. Findings
Table 2 displays the performance (micro and macro ) of all methods on the 7 datasets. WMD scores are not shown for the 20news and wiki datasets, because, after running WMD for 5 days, we stopped experiments with that model on the grounds of its poor efficiency. Due to this, in Table 2 we report the average scores separately for all datasets (allWMD) and for the 5 datasets that WMD was run on (allWMD).
CPTW is the best performing method on both micro and macro on average across all datasets. CPTW is the second best performing method as per micro on all datasets, while WMD is the second best on the 5 datasets it could be run on. With respect to macro , CPTW is the second best performing method on all datasets, and with WMD being the second best when only considering the 5 datasets WMD could be run on. When considering all datasets on macro , then SIF was the best of the baselines, but with both CPTW and CPTW performing better. The worst performing method is LDA, which is most likely because LDA works better on texts with a large number of words. Indeed LDA performs relatively well on wiki – the dataset with the longest texts on average – and noticeably worse than the other methods on most of the other datasets.
4.4. Influence of
We further investigate the stability of the threshold of our model. Specifically, we investigate to what extent (or if at all) optimal values of correlate with a large inter versus intra class ratio in the embedded neighbourhood of a sample document (the intra versus inter class ratio indicates classification difficulty as explained next). We conduct this analysis only with CPTW, our best performing method in Table 2.
We define inter vs intra class ratio (IICR) as the ratio of average Euclidean distances from each point to its closest inter and intra class neighbours. Intra class neighbours refer to the points close to each other within the same class, and inter class neighbours refer to the points close to each other across different classes. The ratio of average Euclidean distances for the inter and intra class neighbours shows how similar the points from one class are compared to the other classes. We study how IICR varies when is changed, and compute IICR as an average over all classes in a dataset for a specific as follows:
[TABLE]
where represents the set of classes, and is the k closest points to from the class. An IICR score close to 1 means that inter and intra class samples are highly similar and thus hard to correctly classify. A higher IICR score than 1 means that inter and intra class samples are more dissimilar, which makes classification fundamentally easier. We consider the closest points, instead of all points, because the space should be transformed such that points become more similar to close intra class points, and become more dissimilar to the closest inter class points. We expect a large value of IICR to correlate well with an appropriate value of for a specific dataset, because we reason that our model should disambiguate semantically related and unrelated documents better. We choose the best performing k on each dataset.
We compute the IICR for all datasets when varying and plot it with varying (Fig. 2), where the legend in the graphs shows the optimal averaged across the cross validations per dataset. Generally, the graphs show that maximum ratios are correlated with values close to the optimal found in the cross validation, except for wiki. This follows our intuition of our model being able to disambiguate the documents such that intra class samples become more similar compared to inter class samples. This means that our approach was able to make documents of the same class more similar when compared to other classes. For wiki most values perform very similar for the best , and when considering the ratios we see a similar trend where all ratios are nearly identical.
5. Conclusion
We presented Contextually Propagated Term Weights, or CPTW that propagate the weight of a word (computed via embeddings) to words occurring in similar contexts. The redistribution of weight has the effect of generalizing the semantics of a text, leading to improved discriminative power. CPTW has low computational cost: state of the art word embeddings are used that can be precomputed efficiently offline, and the propagation itself can be performed efficiently using sparse matrix multiplications based on a precomputed matrix of word similarities.
Experimental evaluation against 8 baselines on 7 well-known datasets shows that CPTW yields the best micro and macro F1 scores on average across all datasets. Most notably, CPTW outperforms strong embedding based methods such as word mover’s distance (WMD) (Kusner et al., 2015) and smooth inverse filtering (SIF) (Arora et al., 2017); likewise, the experiments show that CPTW’s notion of embedded neighbourhood is robust across datasets of different inter vs intra class ratio (i.e. across datasets of increasing classification difficulty). Using the ratio of average euclidean distance between inter and intra class kNN samples as a measure of classification difficulty of a dataset, we found that the value of the (sole) parameter in CPTW chosen by cross validation was close to the maximizing this ratio.
Our work complements recent efforts to employ word embeddings in novel ways that are both (i) computationally efficient, and (ii) semantics-sensitive in that discriminative power is increased by exploiting semantic (Rekabsaz et al., 2016, 2017; Wang et al., 2019) and structural (Hansen et al., 2019) similarities. Along these lines, future research directions include investigating whether, and to what extent, multiple similarity thresholds may improve the overall discriminative power of the method.
Acknowledgements.
Partly funded by Innovationsfonden DK, DABAI (5153-00004A).
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1(1)
- 2Arora et al . (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In ICLR .
- 3Blei et al . (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3, Jan (2003), 993–1022.
- 4Deerwester et al . (1990) Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American society for information science 41, 6 (1990), 391–407.
- 5Hansen et al . (2019) Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. 2019. Neural Speed Reading with Structural-Jump-LSTM. In ICLR .
- 6J et al . (2016) Ganesh J, Manish Gupta, and Vasudeva Varma. 2016. Doc 2Sent 2Vec: A Novel Two-Phase Approach for Learning Document Representation. In SIGIR . ACM, 809–812.
- 7Kenter and De Rijke (2015) Tom Kenter and Maarten De Rijke. 2015. Short text similarity with word embeddings. In CIKM . ACM, 1411–1420.
- 8Kusner et al . (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From Word Embeddings To Document Distances. In ICML . 957–966.
