Learning Concept Embeddings for Efficient Bag-of-Concepts Densification

TL;DR

This paper introduces neural embedding models for concepts to densify Bag-of-Concepts representations, improving text similarity, categorization, and classification tasks with efficient continuous vector aggregation.

Contribution

The paper presents novel neural embedding models and an efficient aggregation method to enhance Bag-of-Concepts representations for various NLP tasks.

Findings

Achieved 1.6% improvement in semantic relatedness correlation

Reduced categorization error rate by over 5%

Outperformed sparse BoC in document classification

Abstract

Explicit concept space models have proven efficacy for text representation in many natural language and text mining applications. The idea is to embed textual structures into a semantic space of concepts which captures the main ideas, objects, and the characteristics of these structures. The so called Bag of Concepts (BoC) representation suffers from data sparsity causing low similarity scores between similar texts due to low concept overlap. To address this problem, we propose two neural embedding models to learn continuous concept vectors. Once they are learned, we propose an efficient vector aggregation method to generate fully continuous BoC representations. We evaluate our concept embedding models on three tasks: 1) measuring entity semantic relatedness and ranking where we achieve 1.6% improvement in correlation scores, 2) dataless concept categorization where we achieve state-of-the-art performance and reduce the categorization error rate by more than 5% compared to five prior word and entity embedding models, and 3) dataless document classification where our models outperform the sparse BoC representations. In addition, by exploiting our efficient linear time vector aggregation method, we achieve better accuracy scores with much less concept dimensions compared to previous BoC densification methods which operate in polynomial time and require hundreds of dimensions in the BoC representation.

Tables

Table 1. Table 1: Top 3 concepts generated using ESA (?) for two 20-newsgroups categories (Hockey and Guns) along with top 3 concepts of sample instances. Using exact match similarity scoring (as in ESA) result in low scores between similar instance and category concept vectors. When using concept embeddings (our models), we obtain relatively higher and more representative similarities.

Category	Top 3 Concepts	Instance Top 3 Concepts	ESA	CCX	CRX
Hockey	- Detroit Red Wings, - History of the Detroit Red Wings, - History of the NHL on United States television	Instance (53798) - History of the Detroit Red Wings, - Detroit Red Wings, - Pittsburgh Penguins	0.73	0.95	0.95
		Instance (54551) - Paul Kariya, - Boston Bruins, - Bobby Orr	0.0	0.84	0.80
Guns	- Waco siege, - Overview of gun laws by nation, - Gun violence in the United States	Instance (54387) - Overview of gun laws by nation, - Waco siege, - Gun politics in the United States	0.71	0.94	0.93
		Instance (54477) - Concealed carry in the United States, - Overview of gun laws by nation, - Gun laws in California	0.33	0.80	0.75

Table 2. Table 2: Example three sentences along with sample contexts generated from CRX and CCX. Contexts are generated with a context window of length 3.

Sentence	CRX Contexts	CCX Contexts
Larry Page is the co-founder of Google which is headquartered in Menlo Park CA	Larry Page, co-founder$>$ co-founder, Google$>$ Google, headquartered$>$ headquartered, Menlo Park CA$>$	Larry Page, Google$>$ Larry Page, Menlo Park CA$>$ Google, Menlo Park CA$>$
Bill Gates is the co-founder of Microsoft which is headquartered in Redmond WA	Bill Gates, co-founder$>$ co-founder, Microsoft$>$ Microsoft, headquartered$>$ headquartered, Redmond WA$>$	Bill Gates, Microsoft$>$ Bill Gates, Redmond WA$>$ Microsoft, Redmond WA$>$
Google is headquartered in Menlo Park CA and was co-founded by Larry Page	Google, headquartered$>$ headquartered, Menlo Park CA$>$ Menlo Park CA, co-founded$>$ co-founded, Larry Page$>$	Google, Menlo Park CA$>$ Google, Larry Page$>$ Menlo Park CA, Larry Page$>$

Table 3. Table 3: Evaluation of concept embeddings for measuring entity semantic relatedness using Spearman rank-order correlation ( ρ 𝜌 \rho ). Overall, the CCX model gives the best results outperforming all other models. It comes 1 st on 3 categories (bold), and 2 nd on the other two (underlined).

Method	IT Companies	Celebrities	TV Series	Video Games	Chuck Norris	All
WLM	0.721	0.667	0.628	0.431	0.571	0.610
CombIC	0.644	0.690	0.643	0.532	0.558	0.624
ExRel	0.727	0.643	0.633	0.519	0.477	0.630
KORE	0.759	0.715	0.599	0.760	0.498	0.698
CRX	0.644	0.592	0.511	0.641	0.495	0.586
CCX	0.788	0.694	0.696	0.708	0.573	0.714

Table 4. Table 4: Top-3 rated entities from CRX & CCX models on sample entities from the 4 domains compared to the ground truth. We can notice high agreement between CCX model ranks and the ground truth ranks (in brackets). The CRX model top rated entities has lower ranks than ground truth ranks causing relatively low correlation scores.

Entity	CRX	CCX	Ground Truth (GT)
Google	Yahoo! (9) Apple Inc. (12) Bing (search engine) (7)	Larry Page (1) Sergey Brin (2) Yahoo! (9)	Larry Page Sergey Brin Google Maps
Leonardo DiCaprio	Kate Winslet (4) Steven Spielberg (9) Tobey Maguire (7)	Tobey Maguire (7) Kate Winslet (4) Titanic (1997 film) (2)	Inception (film) Titanic (1997 film) Frank Abagnale
Mad Men	The Sopranos (15) Matthew Weiner (1) Jon Hamm (2)	Matthew Weiner (1) Jon Hamm (2) Todd London (4)	Matthew Weiner Jon Hamm Alan Taylor (director)
Guitar Hero (video game)	Frequency (video game) (10) Rock Band (video game) (6) Harmonix Music Systems (1)	Harmonix Music Systems (1) WaveGroup Sound (3) RedOctane (1)	Harmonix Music Systems RedOctane WaveGroup Sound

Table 5. Table 5: Accuracy of concept categorization. The CRX model with bootstrapping gives the best results outperforming all other models.

Dataset/Instances	Battig	DOTA-single	DOTA-mult	DOTA-all
Method	(83)	(300)	(150)	(450)
WESenna	0.44	0.52	0.32	0.45
WEMikolov	0.74	0.72	0.67	0.72
TransE1	0.66	0.72	0.69	0.71
TransE2	0.75	0.80	0.77	0.79
TransE3	0.46	0.55	0.52	0.54
CE	0.79	0.89	0.85	0.88
HCE	0.87	0.93	0.91	0.92
CCX	0.72	0.90	0.80	0.87
+boostrap	0.81	0.91	0.85	0.87
CRX	0.83	0.91	0.88	0.90
+bootstrap	0.89	0.98	0.95	0.97

Table 6. Table 6: 20NG category mappings

Top-level	Low-level
Sport	Hockey, Baseball, Autos, Motorcycles
Politics	Guns, Mideast, Misc
Religion	Christian, Atheism, Misc

Table 7. Table 7: Evaluation results of dataless document classification of fine-grained classes measured in micro-averaged F1 along with # of dimensions (concepts) in the BoC at which corresponding performance is achieved.

Method	Hockey x Baseball		Autos x Motorcycles		Guns x Mideast x Misc
ESA	94.60	@425	72.70	@325	70.00	@500
CCX (equal)	94.60	@20	-	-	70.33	@60
CRX (equal)	94.60	@60	73.10	@4	70.00	@7
WEmax	86.85	@65	76.15	@375	72.20	@300
WEhung	95.20	@325	73.75	@300	71.70	@275
CCX (best)	95.10	@125	69.70	@7	72.47	@250
+bootstrap	95.90	@450	74.25	@12	77.43	@5
CRX (best)	95.65	@425	79.20	@14	73.40	@70
+bootstrap	95.90	@350	73.25	@12	77.03	@10

Table 8. Table 8: Evaluation results of dataless document classification of coarse-grained classes measured in micro-averaged F1 along with # of dimensions (concepts) at which corresponding performance is achieved.

Method	Sport x Politics		Sport x Religion
ESA	90.63	@425	94.39	@450
CCX (equal)	92.04	@2	95.11	@6
CRX (equal)	90.99	@2	94.81	@5
WEmax	91.89	@425	93.99	@425
WEhung	90.89	@275	94.16	@450
CCX (best)	92.89	@4	95.86	@60
+bootstrap	93.20	@10	95.13	@225
CRX (best)	93.12	@13	95.91	@95
+bootstrap	92.96	@13	95.53	@70

Table 9. Table 9: Evaluation results of dataless document classification of coarse-grained classes vs. supervised classification with SVM measured in micro-averaged F1 along with # of dimensions (concepts) for CRX & CCX or % of labeld samples for SVM at which corresponding performance is achieved.

Method	Hockey x Baseball		Autos x Motorcycles		Guns x Mideast x Misc
SVM	91.61	@85%	79.25	@20%	77.56	@25%
CCX (best)	95.90	@450	74.25	@12	77.43	@5
CRX (best)	95.90	@350	79.20	@14	77.03	@10

Table 10. Table 10: Top-5 related concepts from CRX & CCX models for sample target concepts

Concept	Concept Raw Context (CRX)	Concept-Concept Context (CCX)
YouTube	Vevo Facebook SoundCloud Vimeo Viral video	Viral video Vimeo Vevo Video blog Dailymotion
Harvard University	Yale University Princeton University Brown University Columbia University Boston University	Harvard Kennedy School Cambridge, Massachusetts Harvard College Radcliffe College Harvard Society of Fellows
Black hole	Neutron star Accretion disk Primordial black hole Supermassive black hole Event horizon	Event horizon Neutron star Gravitational singularity Wormhole Hawking radiation
X-Men: Days of Future Past	X-Men: Apocalypse X-Men: First Class Deadpool (film) Avengers: Age of Ultron X-Men: The Last Stand	X-Men: Apocalypse The Wolverine (film) X-Men: First Class John Paesano William Stryker

Equations

$$\begin{gathered} w_i=\sum _{j=1}^{n}f(c_i,t_j), \\ 1\le i\le N \end{gathered}$$

$$\begin{multlined}Sim_{cos}(\mathbf{u},\mathbf{v})=\frac{\sum_{i=1}^{|\mathbf{u}|}\sum_{j=1}^{|\mathbf{v}|}\mathbbm{1}(n_{i}\mathord{=}m_{j})u_{i}v_{j}}{\sqrt{\sum_{i=1}^{|\mathbf{u}|}u_{i}^{2}}\sqrt{\sum_{j=1}^{|\mathbf{v}|}v_{j}^{2}}}\end{multlined}Sim_{cos}(\mathbf{u},\mathbf{v})=\frac{\sum_{i=1}^{|\mathbf{u}|}\sum_{j=1}^{|\mathbf{v}|}\mathbbm{1}(n_{i}\mathord{=}m_{j})u_{i}v_{j}}{\sqrt{\sum_{i=1}^{|\mathbf{u}|}u_{i}^{2}}\sqrt{\sum_{j=1}^{|\mathbf{v}|}v_{j}^{2}}}$$

$$\frac{1}{V}\sum _{i=1}^V\sum _{-s\le j\le s,j\ne 0}\log p({\omega }_{i+j}|{\omega }_i)$$

$$p({\omega }_O|{\omega }_I)=\frac{\exp ({\mathbf{v}}_{{\omega }_O}^{⊺}{\mathbf{u}}_{{\omega }_I})}{{\sum }_{\omega =1}^W\exp ({\mathbf{v}}_w^{⊺}{\mathbf{u}}_{{\omega }_I})}$$

$$\frac{1}{T}\sum _{i=1}^T\sum _{-s\le j\le s,j\ne 0}\log p(t_{i+j}|t_i)$$

$$\frac{1}{C}\sum _{i=1}^C\sum _{-s\le j\le s,j\ne 0}\log p(c_{i+j}|c_i)$$

$$\log \sigma ({\mathbf{v}}_{{\omega }_O}^{⊺}{\mathbf{u}}_{{\omega }_I})+\sum _{s=1}^k{E}_{{\omega }_s\sim P_n(w)}[\log \sigma (-{\mathbf{v}}_{{\omega }_s}^{⊺}{\mathbf{u}}_{{\omega }_I})]$$

$$\begin{multlined}\mathbf{s}_{dense}=\frac{\sum_{i=1}^{|\mathbf{s}|}w_{i}.\mathbf{u}_{c_{i}}}{\sum_{i=1}^{|\mathbf{s}|}w_{i}}\end{multlined}\mathbf{s}_{dense}=\frac{\sum_{i=1}^{|\mathbf{s}|}w_{i}.\mathbf{u}_{c_{i}}}{\sum_{i=1}^{|\mathbf{s}|}w_{i}}$$

Full text

1 Introduction

Vector space representation models are used to represent textual structures (words, phrases, and documents) as multidimensional vectors. Typically, those models utilize textual corpora and/or Knowledge Bases (KBs) in order to extract and model real-world knowledge. Once acquired, any given text is represented as a vector in the semantic space. The goal is thus to accurately place similar structures close to each other in that semantic space, while placing dissimilar ones far apart.

Explicit concept space models are one of these vector-based representations which are motivated by the idea that, high level cognitive tasks such learning and reasoning are supported by the knowledge we acquire from concepts111A concept is an expression that denotes an idea, event, or an object. (?). Therefore, such models utilize concept vectors (aka bag-of-concepts (BoC)) as the underlying semantic representation of a given text through a process called conceptualization, which is mapping the text into relevant concepts capturing its main ideas, objects, events, and their characteristics. The concept space typically include concepts obtained from KBs such as Wikipedia, Probase (?), and others. Once the concept vectors are generated, similarity between two concept vectors can be computed using a suitable similarity/distance measure such as cosine.

Similar to the traditional Bag-of-Words (BoW) representation, the BoC vector is a multidimensional sparse vector whose dimensionality is the same as the number of concepts in the employed KB (typically millions). Consequently, it suffers from data sparsity causing low similarity scores between similar texts due to low concept overlap. Formally, given a text snippet $T=\{t_{1},t_{2},...,t_{n}\}$ of $n$ terms where $n\geq 1$, and a concept space $C=\{c_{1},c_{2},...,c_{N}\}$ of size $N$. The BoC vector $\mathbf{v}$ = $\{w_{1},w_{2},...,w_{N}\}\in\mathbb{R}^{N}$ of $T$ is a vector of weights of each concept where each $w_{i}$ of concept $c_{i}$ is calculated as in equation 1:

[TABLE]

Here $f(c,t)$ is a scoring function which indicates the degree of association between term $t$ and concept $c$. For example, ? (?) proposed Explicit Semantic Analysis (ESA) which uses Wikipedia articles as concepts and the TF-IDF score of the terms in these article as the association score. Another scoring function might be the co-occurrence count or Pearson correlation score between $t$ and $c$. As we can notice, only very small subset of the concept space would have nonzero scores with a given term222Unless the term is very common (e.g., a, the, some…etc) and carry no relevant information.. Moreover, the BoC vector is generated from the top $n$ concepts which have relatively high association scores with the input terms (typically few hundreds). Thus each text snippet is mapped to a very sparse vector of millions of dimensions having only few hundreds nonzero values leading to the BoC sparsity problem (?).

Typically, the cosine similarity measure is used compute the similarity between a pair of BoC vectors $\mathbf{u}$ and $\mathbf{v}$. Because the concept vectors are very sparse and for space efficiency, we can rewrite each vector as a vector of tuples $(c_{i},w_{i})$. Suppose that $\mathbf{u}\!=\!\{(c_{n_{1}},u_{1}),\dotsc,(c_{n_{|\mathbf{u}|}},u_{|\mathbf{u}|})\!\}$ and $\mathbf{v}\!=\!\{(c_{m_{1}},v_{1}),\dotsc,(c_{m_{|\mathbf{v}|}},v_{|\mathbf{v}|})\!\}$, where $u_{i}$ and $v_{j}$ are the corresponding weights of concepts $c_{n_{i}}$ and $c_{m_{j}}$ respectively. And $n_{i}$, $m_{j}$ are the indices of these concepts in the concept space $C$ such that $1\!\leq\!n_{i},m_{j}\!\leq\!N$. Then, the similarity score can be written as in equation 2:

[TABLE]

where $\mathbbm{1}$ is the indicator function which returns 1 if $n_{i}\mathord{=}m_{j}$ and 0 otherwise. Having such sparse representation and using exact match similarity scoring measure, we can expect that two very similar text snippets might have zero similarity score if they map to different but very related set of concepts (?). We demonstrate this fact in Table 1 (ESA column).

In this paper we utilize neural-based representations to overcome the BoC sparsity problem. The basic idea is to map each concept to a fixed size continuous vector333We use the terms continuous, dense, distributed vectors interchangeably to refer to real-valued vectors.. These vectors can then be used to compute concept-concept similarity and thus overcome the concept mismatch problem.

Our work is also motivated by the success of recent neural-based methods for learning word embeddings in capturing both syntactic and semantic regularities using simple vector arithmetic (?, ?, ?). For example, inferring analogical relationships between words: vec(king)-vec(man)+vec(woman)=vec(queen). This indicates that the learned vectors encode meaningful multi-clustering for each word.

However, word vectors suffer from significant limitations. First, each word is assumed to have a single meaning regardless of its context and thus is represented by a single vector in the semantic space (e.g., charlotte (city) vs. charlotte (given name)). Second, the space contains vectors of single words only. Vectors of multiword expressions (MWEs) are typically obtained by averaging the vectors of individual words. This often produces inaccurate representations especially if the meaning of the MWE is different from the composition of meanings of its individual words (e.g., vec(north carolina) vs. vec(north)+vec(carolina). Additionally, mentions that are used to refer to the same concept would have different embeddings (e.g., u.s., america, usa), and the model might not be able to place those individual vectors in the same sub-cluster, especially the rare surface forms.

We propose two neural embedding models in order to learn continuous concept vectors based on the skip-gram model (?). Our first model is the Concept Raw Context model (CRX) which utilizes raw concept mentions in a large scale textual KB to jointly learn embeddings of both words and concepts. Our second model is the Concept-Concept Context model (CCX) which learns the embeddings of concepts from their conceptual contexts (i.e., contexts containing surrounding concepts only).

After learning the concept vectors, we propose an efficient BoC aggregation method. We perform weighted average of the individual concept vectors to generate fully continuous BoC representations (CBoC). This aggregation method allows measuring the similarity between pairs of BoC in linear time which is more efficient than previous methods that require quadratic or at least log-linear time if optimized (see equation 2). Our embedding models produce more representative similarity scores for BoC containing different but semantically similar concepts as shown in Table 1 (columns 2-3).

We evaluate our embedding models on three tasks:

An intrinsic task of measuring entity semantic relatedness and ranking where we achieve 1.6% improvement in correlation scores. 2. 2.

Dataless concept categorization where we achieve state-of-the-art performance and reduce the categorization error rate by more than 5% compared to five prior word and entity embedding models. 3. 3.

An extrinsic task of dataless document classification which is a learning protocol used to perform text categorization without the need for labeled data to train a classifier (?). Experimental results show that we can achieve better accuracy using our efficient BoC densification method compared to the original sparse BoC representation with much less concept dimensions.

The contributions of this paper are fourfold: First, we propose two low cost concept embedding models which learn concept representations from concept mentions in free-text corpora. Our models require few hours rather than days to train. Second, we show through empirical results the efficacy of the learned concept embeddings in measuring entity semantic relatedness and concept categorization. Our models achieve state-of-the-art performance on two concept categorization datasets. Third, we propose simple and efficient vector aggregation method to obtain fully dense BoC in linear time. Fourth, we demonstrate through experiments on dataless document classification that we can obtain better accuracy using the dense BoC representation with much less dimensions (few in most cases), reducing the computational cost of generating the BoC vector significantly.

The rest of this paper is organized as follows: Section 2 reviews related work; Section 3 describes our proposed embedding models; Section 4 introduces the applications of the proposed models; Section 5 reports experiments and results; and finally, Section 6 provides discussion and concluding remarks.

2 Related Work

Text Conceptualization: Humans understand languages through multi-step cognitive processes which involves building rich models of the world and making multi-level generalizations from the input text (?). One way of automating such generalizations is through text conceptualization. Either by extracting basic level concepts from the input text using concept KBs (?, ?), or mapping the whole input into a concept space that capture its semantics as in ESA (?) and MSA (?).

One major line of conceptualization research utilizes semi-structured KBs such as Wikipedia in order to construct the concept space which is defined by all Wikipedia article titles. Such models have proven efficacy for semantic analysis of textual data especially short texts where contextual information is missing or insufficient (see ? (?) for examples).

Another research direction uses more structured concept KBs such as Probase444https://concept.research.microsoft.com (?). Probase is a probabilistic KB of millions concepts and their relationships (basically is-a). It was created by mining billions of Web pages and search logs of Microsoft’s Bing555https://www.bing.com/ repository using syntactic patterns. The concept KB was then leveraged for text conceptualization to support various text understanding tasks such as clustering of Twitter messages and News titles (?, ?), (?), search query understanding (?), short text segmentation (?, ?), and term similarity (?, ?).

Despite its effectiveness, the dependency of Probase on syntactic patterns can be a limitation especially for languages other than English. In addition, we expect augmenting and maintaining these syntactic patterns to be costly and labor intensive. We argue that concept embeddings allow simpler and more efficient representations, simply because similarity scoring between concept vectors can be performed using vector arithmetic. While the Probase hierarchy allows only symbolic matching which still suffers data sparsity. On another hand, we spotted some cases where Probase probabilities were atypical666$p(\mbox{Arabic coffee}\mid\mbox{beverage})=0$. This is due to learning concept categories from a limited set of syntactic patterns which does not cover all concept mention patterns. Concept embeddings relax this problem by leveraging all concept mentions in order to learn the embedding vector and therefore might be utilized to curate such atypical Probase assertions.

Concept/Entity Embeddings: Neural embedding models have been proposed to learn distributed representations of concepts/entities777In this paper we use the terms ”concept” and ”entity” interchangeably.. ? (?) proposed using the popular Word2Vec model (?) to obtain the embeddings of each concept by averaging the embeddings of the concept’s individual words. For example, the embeddings of Microsoft Office would be obtained by averaging the vector of Microsoft and the vector of Office obtained from the Word2Vec model. Clearly, this method disregards the fact that the semantics of multiword concepts whose composite meaning is different from the semantics of their individual words.

More robust concept and entity embeddings can be learned from the general knowledge about the concept in encyclopedic KB (e.g., its article) and/or from the structure of a hyperlinked KB (e.g., its link graph). Such concept embedding models were proposed by ? (?, ?), and ? (?) who all utilize the skip-gram learning technique (?), but differ in how they define the context of the target concept.

? (?) extended the embedding model proposed by ? (?) by jointly learning entity and category embeddings from contexts defined by all other entities in the target entity article as well as its category hierarchy in Wikipedia. This method has the advantage of learning embeddings of both entities and categories jointly. However, defining the entity contexts as pairs of the target entity and all other entities appearing in its corresponding article might introduce noisy contexts, especially for long articles. For example, the Wikipedia article for ”United States” contains links to ”Kindergarten”, ”First grade”, and ”Secondary school” under the ”Education” section.

? (?) proposed a method based on the skip-gram model to jointly learn embeddings of words and entities using contexts generated from surrounding words of the target entity or word. The authors also proposed incorporating Wikipedia link graph by generating contexts from all entities with outgoing link to the target entity to better model entity-entity relatedness.

Our models also learn word and concept embeddings jointly. Mapping both words and concepts into the same semantic space allows us to easily measure word-word, word-concept, and concept-concept semantic similarities. In addition, our CRX model (described in Section 3) extends the context of each word/concept by including nearby concept mentions and not only nearby words. Therefore, we better model local contextual information of concepts and words in Wikipedia, treated as a textual KB. During training, we generate word-word, word-concept, concept-word, and concept-concept contexts (cf. equation 5). In ? (?) model, concept-concept contexts are generated from Wikipedia link graph not from their raw mentions in Wikipedia text. In the CCX model, we define concept contexts by all surrounding concepts within a window of fixed size.

Generating contexts from raw text mentions makes our models scalable and not restricted to hyperlinked encyclopedic textual corpora only. This facilitates exploiting other free-text corpora with annotated concept mentions (e.g., news stories, scientific publications, medical guidelines…etc). Moreover, our proposed models are computationally less costly than ? (?) and ? (?) models as they require few hours rather than days to train on similar computing resources.

BoC Densification: Densification of the Bag-of-Concepts (BoC) is the process of converting the sparse BoC into a continuous BoC (CBoC) (aka dense BoC) in order to overcome the BoC sparsity problem. The process requires first mapping each concept into a continuous vector using representation learning. ? (?) proposed three different mechanisms for aligning the concepts at different indices given a sparse BoC pair ($\mathbf{u},\mathbf{v}$) in order to increase their similarity score.

The many-to-many mechanism works by averaging all pairwise similarities. The many-to-one mechanism works by aligning each concept in $\mathbf{u}$ with the most similar concept in $\mathbf{v}$ (i.e., its best match). Clearly, the complexity of these two mechanisms is quadratic. The third mechanism is the one-to-one. It utilizes the Hungarian method in order to find an optimal alignment on a one-to-one basis (?). This mechanism performed the best on the task of dataless document classification and was also utilized by ? (?).

However, the Hungarian method is a combinatorial optimization algorithm whose complexity is polynomial. Our proposed densification mechanism is more efficient than these three mechanisms as its complexity is linear with respect to the number of nonzero elements in the BoC. Additionally, it is simpler as it does not require tuning a cutoff threshold for the minimum similarity between two aligned concepts as in previous work. Figure 1 shows a schematic diagram of our efficient densification mechanism applied to a BoC generated from a Wikipedia inverted index. We simply perform weighted average of the individual concept vectors in the obtained BoC where concept weights correspond to the TF-IDF scores from searching Wikipedia.

3 Learning Concept Embeddings

A main objective of learning concept embeddings is to overcome the inherent problem of data sparsity associated with the BoC representation. Here we try to learn continuous concept vectors by building upon the skip-gram embedding model (?). In the conventional skip-gram model, a set of contexts are generated by sliding a context window of predefined size over sentences of a given text corpus. Vector representation of a target word is learned with the objective to maximize the ability of predicting surrounding words of that target word.

Formally, given a training corpus of $V$ words $\omega_{1},\omega_{2},...,\omega_{V}$. The skip-gram model aims to maximize the average log probability:

[TABLE]

where $s$ is the context window size, $\omega_{i}$ is the target word, and $\omega_{i+j}$ is a surrounding context word. The softmax function is used to estimate the probability $p(\omega_{O}|\omega_{I})$ as follows:

[TABLE]

where $\mathbf{u}_{\omega_{I}}$ and $\mathbf{v}_{\omega_{O}}$ are the input and output vectors respectively and $W$ is the vocabulary size. ? (?) proposed hierarchical softmax and negative sampling as efficient alternatives to approximate the softmax function which becomes computationally intractable when $W$ becomes huge.

Our approach genuinely learns distributed concept representations by generating concept contexts from mentions of those concepts in large encyclopedic KBs such as Wikipedia. Utilizing such annotated KBs eliminates the need to manually annotate concept mentions and thus comes at no cost.

3.1 Concept Raw Context Model (CRX)

In this model, we jointly learn the embeddings of both words and concepts. First, all concept mentions are identified in the given corpus. Second, contexts are generated for both words and concepts from other surrounding words and other surrounding concepts as well. After generating all the contexts, we use the skip-gram model to jointly learn the embeddings of words and concepts. Formally, given a training corpus of $V$ words $\omega_{1},\omega_{2},...,\omega_{V}$, we iterate over the corpus identifying words and concept mentions and thus generating a sequence of $T$ tokens $t_{1},t_{2},...t_{T}$ where $T<V$ (as multiword concepts will be counted as one token). Afterwards we train the a skip-gram model aiming to maximize:

[TABLE]

where as in the conventional skip-gram model, $s$ is the context window size. In this model, $t_{i}$ is the target token which would be either a word or a concept mention, and $t_{i+j}$ is a surrounding context word or concept mention.

This model is different from ? (?)’s anchor context model in three aspects: 1) while generating target concept contexts, we utilize not only surrounding words but also other surrounding concepts, 2) our model aims to maximize $p(t_{i+j}|t_{i})$ where $t$ could be a word or a concept, while ? (?) model maximizes $p(\omega_{i+j}|e_{i})$ where $e_{i}$ is the target concept/entity (see ? (?) Eq. 6), and 3) in case $t_{i}$ is a concept, our model captures all the contexts in which it appeared, while ? (?) model generates for each entity one context of $s$ previous and $s$ next words. We hypothesize that considering both concepts and individual words in the optimization function generates more robust embeddings.

3.2 Concept-Concept Context Model (CCX)

Inspired by the distributional hypothesis (?), in this model, we hypothesize that: ”similar concepts tend to appear in similar conceptual contexts”. In order to test this hypothesis, we propose learning concept embeddings by training a skip-gram model on contexts generated solely from concept mentions. As in the CRX model, we start by identifying all concept mentions in the given corpus. Then, contexts of target concept are generated from surrounding concepts only. Formally, given a training corpus of $V$ words $\omega_{1},\omega_{2},...,\omega_{V}$. We iterate over the corpus identifying concept mentions and thus generating a sequence of $C$ concept tokens $c_{1},c_{2},...c_{C}$ where $C<V$. Afterwards we train the skip-gram model aiming to maximize:

[TABLE]

where $s$ is the context window size, $c_{i}$ is the target concept, and $c_{i+j}$ is a surrounding concept mention within $s$ mentions.

This model is different from ? (?) and ? (?) as they define the context of a target concept by all the other concepts which have an outgoing link from the concept’s corresponding article in Wikipedia.

Formally, given an article about concept $c_{t}$ containing other concepts ($c_{1},c_{2},...,c_{n}$), ? (?, ?) create context pairs in the form ($c_{t}$,$c_{i}$), $1\leq i\leq n$. Thus context size is limited to only 2 concepts.

Clearly, some of these concepts might be irrelevant especially for very long articles which cite hundreds of other concepts. Our CCX model, alternatively, learns concept semantics from surrounding concepts and not only from those that are cited in its article. We also extend the context window beyond pairs of concepts (based on $s$ in equation 6) allowing more influence to other nearby concepts.

3.3 CRX vs. CCX

One of the advantages of the CCX model over the CRX model is its computational efficiency during learning. On the other hand, the CCX model vocabulary is limited to the corpus concepts (all Wikipedia articles in our case), while the CRX model vocabulary is defined by all unique concepts+words in Wikipedia.

Another distinct property of the CCX model is its emphasis on concept-concept relatedness rather than similarity (as we will detail more in the experiments section). The CCX model by looking only at surrounding concept mentions while learning, is able to generate contexts containing more diverse but related concepts. One the other hand, the CRX model which jointly learns the embeddings of words and concepts puts more emphasis on similarity by leveraging the full contextual information of words and concepts while learning.

To better illustrate this difference, consider a sample of the contexts generated from CRX and CCX in Table 2 using a sliding window of length 3. As we can notice, the CRX contexts of ”Google” and ”Microsoft” are somewhat similar containing words like ”headquartered” and ”co-founder”. This causes the model to learn similar vectors for these two concepts. On the other hand, the CCX contexts of ”Google” and ”Microsoft” do not share any similarities888This is an illustrative example and doesn’t imply the two concepts will have totally dissimilar vectors., rather we can see that ”Google” has similar contexts to ”Larry Page” as both has ”Menlo Park CA” in their contexts, causing the model to learn similar embeddings for these two related concepts.

3.4 Training

We utilize a recent Wikipedia dump of August 2016999http://dumps.wikimedia.org/enwiki/, which has about 7 million articles. We extract articles plain text discarding images and tables. We also discard ”References” and ”External links” sections (if any). We pruned both articles not under the main namespace and pruned all redirect pages as well. Eventually, our corpus contained about 5 million articles in total.

We preprocess each article replacing all its references to other Wikipedia articles with the their corresponding page IDs. In case any of the references is a title of a redirect page, we use the page ID of the original page to ensure that all concept mentions are normalized.

Following ? (?), we utilize negative sampling to approximate the softmax function by replacing every $\log\ p(\omega_{O}|\omega_{I})$ term in the softmax function (equation 4) with:

[TABLE]

where $k$ is the number of negative samples drawn for each word and $\sigma(x)$ is the sigmoid function ($\frac{1}{1+e^{-x}}$). In the case of the CRX model $\omega_{I}$ and $\omega_{O}$ would be replaced with $t_{i}$ and $t_{i+j}$ respectively. And in the case of the CCX model $\omega_{I}$ and $\omega_{O}$ would be replaced with $c_{i}$ and $c_{i+j}$ respectively.

For both the CRX & CCX models with use a context window of size 9 and a vector of 500 dimensions. We train the skip-gram model for 10 iterations using 12-core machine with 64GB of RAM. The CRX model took $\sim$15 hours to train for a total of $\sim$12.7 million tokens. The CCX model took $\sim$1.5 hours to train for a total of $\sim$4.5 million concepts.

3.5 BoC Densification

As we mentioned in the related work section, the current mechanisms for BoC densification are inefficient as their complexity is at least quadratic with respect to the number of nonzero elements in the BoC vector. Here, we propose simple and efficient vector aggregation method to obtain fully continuous BoC vectors (CBoC) in linear time. Our mechanism works by performing a weighted average of the individual concept vectors in a given BoC. This operation has two advantages. First, it scales linearly with the number of nonzero dimensions in the BoC vector. Secondly, it produces a fully dense BoC vector of fixed size representing the semantics of the original concepts and considering their weights. Formally, given a sparse BoC vector $\mathbf{s}\!=\!\{(c_{1},w_{1}),\dotsc,(c_{|\mathbf{s}|},w_{|\mathbf{s}|})\!\}$, where $w_{i}$ is weight of concept $c_{i}$101010The weights are the TF-IDF scores from searching Wikipedia.. We can obtain the dense representation of $\mathbf{s}$ as in equation 8:

[TABLE]

where $\mathbf{u}_{c_{i}}$ is the vector of concept $c_{i}$. Once we have this dense BoC vector, we can apply the cosine measure to compute the similarity between a pair of dense BoC vectors.

As we can notice, this weighted average is done once for any given BoC vector. Other mechanisms that rely on concept alignment (?), require realignment every time a pair of BoC vectors are compared. Our approach improves the efficiency especially in the context of dataless document classification with large number of classes. Using our densification mechanism, we apply the weighted average for the BoC of each category and for each instance document once.

Interestingly, our densification mechanism allows us to densify the sparse BoC vector using only the top few dimensions. As we will show in the experiments, we can get near-best results using these few dimensions compared to densifying with all the dimensions in the original sparse vector. This property reduces the cost of obtaining a BoC vector with a few hundred dimensions in the first place.

4 Text Conceptualization Applications

Concept-based representations have many applications in computational linguistics, information retrieval, and knowledge modeling. Such representations are able to capture the semantics of a given text by either identifying concept mentions in that text, transforming the text into a concept space, or both (?). Thereafter, many cognitive tasks that require huge background and real-world knowledge are facilitated by leveraging the conceptual representations. We describe some of these tasks in this section, and provide empirical evaluation of our our concept embedding models on such tasks in the next section.

4.1 Concept/Entity Relatedness

Entity relatedness has been recently used to model entity coherence in many named entity linking and disambiguation systems (?, ?, ?, ?, ?, ?, ?). In entity search, ? (?) utilized entity relatedness score to rank candidate entities based on their relatedness to the search query entities. Also, entity embeddings have proved more efficient and effective for measuring entity relatedness over traditional relatedness measures which use link analysis. Formally, given a entity pair ($e_{i},e_{j}$), their relatedness score is evaluated as $rel(e_{i},e_{j})=Sim(\mathbf{u}_{e_{i}},\mathbf{u}_{e_{j}})$, where $Sim$ is a similarity function (e.g., cosine), and $\mathbf{u}_{e}$ is the embeddings of entity $e$.

4.2 Concept Learning

Concept learning is a cognitive process which involves classifying a given concept/entity to one or more candidate categories (e.g., milk as beverage, dairy product, liquid…etc). This process is also known as concept categorization111111In this paper, we use concept learning and concept categorization interchangeably (?). Automated concept learning gains its importance in many knowledge modeling tasks such as knowledge base construction (discovering new concepts), completion (inferring new relationships between concepts), and curation (removing noisy or assessing weak relationships). Similar to ? (?), we assign a given concept to a target category using Rocchio classification (?), where the centroid of each category is set to the category’s corresponding embedding vector. Formally, given a set of $n$ candidate concept categories $G=\{g_{1},...,g_{n}\}$, a sample concept $c$, an embedding function $f$, and a similarity function $Sim$, then $c_{i}$ is assigned to category $g_{*}$ such that $g_{*}=arg\ \mbox{max}_{i}\ Sim(f(g_{i}),f(c))$. Here, the embedding function $f$ would always map the given concept to its vector.

4.3 Dataless Classification

? (?) proposed dataless document classification as a learning protocol to perform text categorization without the need for labeled data to train a classifier. Given only label names and few descriptive keywords of each label, classification is performed on the fly by mapping each label into a BoC representation using ESA (?). Likewise, each data instance is mapped into the same BoC semantic space and assigned to the most similar label using a proper similarity measure such as cosine. Formally, given a set of $n$ labels $L=\{l_{1},...,l_{n}\}$, a text document $d$, a BoC mapping model $f$, and a similarity function $Sim$. First we conceptualize the each $l_{i}$ and the document $d$ by applying $f$ on them, which will produce sparse BoC vectors $s_{l_{i}}$ and $s_{d}$ respectively. Then we densify the vectors as in equation 8 producing $s_{dense_{{l_{i}}}}$ and $s_{dense_{d}}$ respectively. Finally $d$ is assigned to label $l_{*}$ such that $l_{*}=arg\ \mbox{max}_{i}\ Sim(s_{dense_{{l_{i}}}},s_{dense_{d}})$.

In the context of dataless classification, ? (?) and ? (?) used bootstrapping in order to improve the classification performance without the need for labeled data. The basic idea is to start from target labels as the initial training samples, train a classifier, and iteratively add to the training data those samples which the classifier is most confident until no more samples to be classified. The results of dataless classification with bootstrapping were competitive to supervised classification with many training examples.

In this paper, we extend the use of bootstrapping to the concept learning task as well. In concept learning we start with the vectors of target category concepts as a prototype view upon which categorization decisions are made (e.g., vec(bird), vec(mammal)…etc). We leverage bootstrapping by iteratively updating this prototype view with the vectors of concept instances we are most confident. For example, if ”deer” is closest to ”mammal” than any other instance in the dataset, then we update the definition of ”mammal” by performing vec(mammal)+=vec(deer), and repeat the same operation for other categories as well. This way, we adapt the initial prototype view to better match the specifics of the given data. Although bootstrapping is a time consuming process, we argue that, using dense vectors for representing concepts makes bootstrapping more appealing. As updating the category vector with an instance vector could be performed through optimized vector arithmetic which is available in most modern machines. Algorithm 1 presents the pseudocode for performing dataless classification and concept categorization with bootstrapping. In our implementation, we bootstrap the category vector with vectors of the most similar $\mathbf{N}$ instances at a time. Another implementation option might be defining a threshold and bootstrapping using vectors of $\mathbf{N}$ instances if their similarity score exceed that threshold. In the experiments, we set $\mathbf{N}\!=\!1$.

5 Experiments

5.1 Entity Semantic Relatedness

We evaluate the ”goodness” of our concept embeddings on measuring entity semantic relatedness as an intrinsic evaluation.

5.1.1 Dataset

We use the KORE dataset created by ? (?). It consists of 21 main entities from four domains: IT companies, Hollywood celebrities, video games, and television series. For each of these entities, 20 other candidate entities were selected and manually ranked based on their relatedness score based on human judgements. As in previous studies, we report the Spearman rank-order correlation ($\rho$) (?) which assesses how the automated ranking of candidate entities based on their relatedness score matches the ranking we obtain from human judgements.

5.1.2 Compared Systems

We compare our models with four previous methods:

KORE (?) which measure entity relatedness by firstly representing entities as sets of weighted keyphrases and then computing relatedness using different measures such as keyphrase vector cosine similarity and keyphrase overlap relatedness. 2. 2.

WLM introduced by ? (?) who proposed a Wikipedia Link-based Measure (WLM) as a simple mechanism for modeling the semantic relatedness between Wikipedia entities. The authors utilized Wikipedia link structure under the assumption that related entities would have similar incoming links. 3. 3.

Exclusivity-based Relatedness (ExRel) introduced by ? (?) who proposed this measure under the assumption that not all instances of a given relation type should be equally weighted. Specifically, the authors hypothesized that the relatedness score between two concepts should be higher if each of them is related through the same relation type to fewer other concepts in the employed KB link graph. 4. 4.

Combined Information Content (CombIC) introduced by ? (?) who compute the relatedness score using a graph edit distance measure on the DBpedia KB.

5.1.3 Results

Table 3 shows the Spearman ($\rho$) correlation scores of the CRX and CCX model compared to previous models. As we can notice the CCX model achieves the best overall performance on the five domains combined exceeding its successor KORE by 1.6%. The CRX model on the other hand came last on this task.

In order to better understand these results, we looked at rankings of individual entities from each domain to see how they compare to the ground truth. Table 4 shows the top-3 rated entities from each model on sample entities from the four domains. As we can notice, the ground truth assigns high rank to related rather than similar entities. For example, relatedness of ”Google” to ”Larry Page” is ranked 1st, while to ”Yahoo!” is ranked 9th, and to ”Apple Inc.” is ranked 12th. As the CCX model emphasizes semantic relatedness over similarity, it has high overlap in the top-3 entities with the ground truth (underlined entities). On the other hand, the CRX model predictions are actually meaningful when it comes to functional and topical similarity. As we can notice, it assigns high ranks of ”Google” to other companies (”Yahoo!”, ”Apple Inc.”), of ”Leonardo DiCaprio” to other celebrities (”Tobey Maguire”), and ”Mad Men” to other TV series (”The Sopranos”), and of ”Guitar Hero” to other video games (”Frequency”, ”Rock Band”). However, all these highly ranked entities by CRX have relatively low rankings in the ground truth (given in brackets). This caused the correlation score to be much lower than what we obtained from the CCX model.

The results indicate that, the CCX model could be more appropriate in applications where relatedness and topical diversity are more desired than topical and functional coherence where the CRX model would be more appealing.

5.2 Concept Categorization

This task can be viewed as both intrinsic and extrinsic. It is intrinsic because a good embedding model would generate clusters of concepts belonging to the same category, and optimally place the category vector at the center of its instances vectors. On another hand, it is extrinsic as the embedding model could be used to generate a concept KB of is-a relationships with confidence scores, similar to Probase (?). The model could even be used to curate and/or assert the facts in Probase.

5.2.1 Datasets

As in ? (?), we utilize two benchmark datasets: 1) Battig test (?), which contains 83 single word concepts (e.g., cat, tuna, spoon..etc) belonging to 10 categories (e.g., mammal, fish, kitchenware..etc), and 2) DOTA which was created by ? (?) from Wikipedia article titles (entities) and category names (categories). DOTA contains 300 single-word concepts (DOTA-single) (e.g., coffee, football, semantics..etc), and (150) multiword concepts (DOTA-mult) (e.g., masala chai, table tennis, noun phrase…etc). Both belong to 15 categories (e.g., beverage, sport, linguistics..etc). Performance is measured in terms of the ability of the system to assign concept instances to their correct categories.

5.2.2 Compared Systems

We compare our models to various word, entity, and category embedding methods as described in ? (?) including:

Word embeddings**:** ? (?) model (WESenna) trained on Wikipedia. Here vectors of multiword concepts are obtained by averaging their individual word vectors. 2. 2.

MWEs embeddings**:** ? (?) model (WEMikolov) trained on Wikipedia. This model jointly learns single and multiword embeddings where MWEs are identified using corpus statistics. 3. 3.

Entity-category embeddings**:** which include ? (?) embedding model (TransE). This model utilizes relational data between entities in a KB as triplets in the form (entity,relation,entity) to generate representations of both entities and relationships. ? (?) implemented three variants of this model (TransE1, TransE2, TransE3) to generate representations for entities and categories jointly. Two other models introduced by ? (?) are CE and HCE. CE generates embeddings for concepts and categories using category information of Wikipedia articles. HCE extends CE by incorporating Wikipedia’s category hierarchy while training the model to generate concept and category vectors.

5.2.3 Results

We report the accuracy scores of concept categorization121212From a multi-class classification perspective, the accuracy scores would be equivalent to the clustering purity score as reported in ? (?). in Table 5. Accuracy is calculated by dividing the number of correctly classified concepts by the total number of concepts in the given dataset. Scores of all other methods are obtained from ? (?). As we can see in Table 5, the CRX model comes second after the HCE on all datasets. While the CCX model performance is much less than CRX. With bootstrapping, the CCX model performance improves on both datasets. CRX with bootstrapping outperforms all other models by significant percentages. These results show that learning concept embeddings from concept mentions is actually different from training the skip-gram model on phrases or multiword expressions. This is clear from the significant performance gains we get from the CRX and CCX models compared to WEMikolov which was trained using skip-gram on phrases. Additionally, the results demonstrate the efficacy of our models which simply learn concept embeddings from concept mentions in free-text corpus compared to the more complex models which require category or relational information such as TransE, CE, and HCE.

5.3 Dataless Classification

In this experiment, we evaluate the effectiveness of our concept embedding models on the dataless document classification task as an extrinsic evaluation. We demonstrate through empirical results the efficiency and effectiveness of our proposed BoC densification scheme which helps obtaining better classification results compared to the original sparse BoC representation.

5.3.1 Dataset

We use the 20-newsgroups dataset (20NG) (?) which is commonly used for benchmarking text classification algorithms. The dataset contains 20 categories each has $\sim$1000 news posts. We obtained the BoC representations using ESA from ? (?) who utilized a Wikipedia index containing pages with 100+ words and 5+ outgoing links to create ESA mappings of 500 dimensions for both the categories and news posts of the 20NG. We designed two types of classification tasks: 1) fine-grained classification involving closely related classes such as Hockey vs. Baseball, Autos vs. Motorcycles, and Guns vs. Mideast vs. Misc, and 2) coarse-grained classification involving top-level categories such as Sport vs. Politics and Sport vs. Religion. The top-level categories are created by combining instances of the fine-grained categories which are shown in Table 6.

5.3.2 Compared Systems

We compare our models to three previous methods:

ESA which computes the cosine similarity between target labels and instance documents using the sparse BoC vectors. 2. 2.

WEmax & WEhung which were proposed by ? (?) for BoC densification using embeddings obtained from Word2Vec. As the authors reported, we fix the minimum similarity threshold to 0.85. WEmax finds the best match for each concept, while WEhung utilizes the Hungarian algorithm to find the best concept-concept alignment on one-to-one basis. Both mechanisms have polynomial degree time complexity.

5.3.3 Results

Table 7 presents the results of fine-grained dataless classification measured in micro-averaged F1. As we can notice, ESA achieves its peak performance with a few hundred dimensions of the sparse BoC vector. Using our densification mechanism, both the CRX & CCX models achieve equal performance to ESA at many fewer dimensions. Densification using the CRX model embeddings gives the best F1 scores on the three tasks. Interestingly, the CRX model improves the F1 score by $\sim$7% using only 14 concepts on Autos vs. Motorcycles, and by $\sim$3% using 70 concepts on Guns vs. Mideast vs. Misc. The CCX model, still performs better than ESA on 2 out of the 3 tasks. Both WEmax and WEhung improve the performance over ESA but not as our CRX model.

When we applied bootstrapping, the performance of the CCX model improved slightly on Hockey vs. Baseball, but significantly ($\sim$5%) on the other two tasks achieving best performance on the third task with just 5 concepts. Bootstrapping with the CRX model has a similar effect to the CCX model except for Autos vs. Motorcycles where performance degraded significantly. To better understand this behavior, we analyzed the results as bootstrapping progresses at 14 concepts like CRX (best). We noticed that, at the very early iterations of Algorithm 1, many instances belonging to Autos were closer to Motorcycles with similarity scores between 0.90-0.95. And when using those instances to bootstrap Motorcycles, they caused topic drift moving Motorcycles’s centroid toward Autos, and eventually causing relatively lower accuracy scores.

In order to better illustrate the robustness of our densification mechanism when varying the number of BoC dimensions, we measured F1 scores of each task as a function of the number of BoC dimensions used for densification. As we see in Figure 2, with one concept we can achieve high F1 scores compared to ESA which achieves zero or very low score. Moreover, near-peak performance is achievable with the top 50 or less dimensions. We can also notice that, as we increase the number of dimensions, both WEmax and WEhung densification methods have the same undesired monotonic pattern like ESA. Actually, the imposed threshold by these methods does not allow for full dense representation of the BoC vector and therefore at low dimensions we still see low overall F1 score. Our proposed densification mechanism besides its low cost, produces fully densified representations allowing good similarities at low dimensions.

Results of coarse-grained classification are presented in Table 8. Classification at the top level is easier than the fine-grained level. Nevertheless, as with fine-grained classification, ESA still peaks with a few hundred dimensions of the sparse BoC vector. Both the CRX & CCX models achieve equal performance to ESA at very few dimensions ($\leq\!6$). Densification using the CRX model embeddings still performs the best on both tasks. Interestingly, the CCX model gives very close F1 scores to the CRX model at less dimensions (@4 with Sport vs. Politics, and @60 with Sport vs. Religion) indicating its competitive advantage when training computational cost is a decisive criteria. The CCX model, still performs better than ESA, WEmax, and WEhung on both tasks.

Bootstrapping did not improve the results on this task significantly (if any). As we can notice in Table 8, the accuracy without bootstrapping is already high indicating that the initial prototype vector (centroid) of each class is representative enough of the instances to be classified.

Figure 3 shows F1 scores of coarse-grained classification when varying the # of BoC dimensions used for densification. The same pattern of achieving near-peak performance at very few dimensions recur with the CRX & CCX models. ESA using the sparse BoC vectors achieves low F1 up until few hundred dimensions are considered. Even with the costly WEmax and WEhung densifications, performance sometimes decreases.

5.3.4 Dataless vs. Supervised Classification

We performed a pilot experiment to demonstrate the value of the dataless classification scheme in the absence or difficulty of obtaining labeled data for training a supervised classifier. For this purpose, we used a Support Vector Machine (SVM) classifier with a linear kernel, leveraging the scikit-learn machine learning library (?) to perform classification of fine-grained classes (cf. Table 6). We trained the SVM classifier with labeled samples ranging between 10% to 90% of the total number of samples for each task and evaluate the performance on the rest. The results in Table 9 shows the % of labeled sampled needed for training SVM in order to achieve equal performance to dataless classification with CRX and CCX. As we can notice, with Hockey vs Baseball, the SVM classifier can’t reach the same performance as either models and peaks when trained on 85% ($\sim$1700 samples) of the data. With the Autos vs Motorcycles and Guns vs Mideast vs Misc, the SVM classifier achieves equal performance when trained on 20% ($\sim$400 samples) and 25% ($\sim$750 samples) of the data respectively. These results demonstrate the competitiveness of our models to supervised classification even when training data is available. And its superiority when training data is scarce.

6 Discussion & Conclusion

In this paper we proposed two models for learning neural embeddings of explicit concepts based on the skip-gram model. Explicit concepts are lexical expressions (single or multiwords) that denote an idea, event, or an object and typically have a set of properties associated with it. In the models presented here, our concept space is the set of all Wikipedia article titles. We proposed learning concept representations from concept mentions/references in Wikipedia making our models applicable to other open domain and domain specific free-text corpora by firstly wikifying131313Wikification is the process of identifying mentions of concepts and entities in a given free-text and linking them to Wikipedia the text and then learning from concept mentions.

It is clear from the presented results that, the CRX model outperforms the CCX model on tasks that require topical coherence among the concepts vectors (e.g. concept categorization), while the CCX model is advantageous in tasks that require topical relatedness (e.g., measuring entity relatedness). To better show this difference qualitatively, we present a qualitative analysis of both models in Table 10 (target concepts are similar to those reported by ? (?)).

As we can notice, the CRX model tends to emphasize concept topical and categorical similarity, while the CCX model tends to more emphasize concept relatedness. For example, the top-5 concepts closest to ”Harvard University” using CRX are all universities. While, the CCX model top-5 concepts include, besides educational institutions, location (”Cambridge, Massachusetts”) and an affiliated group (”Harvard Society of Fellows”). The same pattern can be noticed with the ”X-Men” movie where we get similar genre movies with CRX. While we get related characters such as *”William Stryker”*141414https://en.wikipedia.org/wiki/William_Stryker with CCX.

Based on these observations, we claim that the CCX model would be beneficial in situations where diversity is more desired than topical coherence. This claim is also supported by the results we obtained on the concept categorization and dataless densification tasks. On concept categorization, the performance gap between CRX and CCX was large with almost all datasets. On dataless classification, the performance gap was large with documents belonging topics with nuance differences (i.e., Autos vs. Motorcycles), but with other classes which have clear distinctions, the CCX performance was very competitive to CRX (e.g., Hockey vs Baseball).

In this paper, we also proposed an efficient and effective mechanism for BoC densification which outperformed the previously proposed densification schemes on dataless document classification. Unlike these previous densification mechanisms, our method scales linearly with the number of the BoC dimensions. In addition, we demonstrated through the results how this efficient mechanism allows generating high quality dense BoC from few concepts alleviating the need of obtaining hundreds of concepts when generating the BoC in the first place.

Our learning method does not require training on a hierarchical concept category graph and is not tightly coupled to linked knowledge base. Rather, we learn concept representations using mentions in free-text corpora with annotated concept mentions which even if not available could be obtained through state-of-the-art entity linking systems.

Finally, the work presented in this paper serves two of our objectives: 1) it demonstrates utilizing textual knowledge bases to learn robust concept embeddings and hence increasing the effectiveness of the BoC representation to better capture semantic similarities between textual structures, and 2) it demonstrates utilizing the learned distributed concept vectors to increase the efficiency of the semantic representations in terms of space and computational complexities.