Characterization of citizens using word2vec and latent topic analysis in a large set of tweets

TL;DR

This paper presents a machine learning approach using word2vec and latent topic analysis to characterize city communities based on a large dataset of tweets, revealing insights into citizen ideas and community structures.

Contribution

It introduces a novel method combining word embeddings and topic modeling to automatically detect and analyze city communities from social media data.

Findings

Effective characterization of city populations using tweets

Identification of community structures through text analytics

Demonstrated approach on a large dataset of over 2.6 million tweets

Abstract

With the increasing use of the Internet and mobile devices, social networks are becoming the most used media to communicate citizens' ideas and thoughts. This information is very useful to identify communities with common ideas based on what they publish in the network. This paper presents a method to automatically detect city communities based on machine learning techniques applied to a set of tweets from Bogot\'a's citizens. An analysis was performed in a collection of 2,634,176 tweets gathered from Twitter in a period of six months. Results show that the proposed method is an interesting tool to characterize a city population based on a machine learning methods and text analytics.

Tables

Table 1. Table 1: The most frequent words of each cluster. Each cluster is represented by 3 cells: the first one tells the number of the cluster (#) and the topic (in parentheses, if the topic is easily identified); the second one tells the average number of characters per tweet of that cluster; and the third one, tells the 10 most common words of the cluster (with their respective translations in English in parentheses) for the 15 clusters with the larger tweet length average, and 6 most common words for the rest of the clusters.

#22 (News)	104	#30 (Country Politics)	87	#34 (Sports)	87	#24 (News)	86	#25 (Country Politics)	85
hoy (today)		Santos (Colombia’s president)		gol (goal)		via (way)		Santos (Colombia’s president)
Tunja (a city)		FARC (Colombian guerrilla)		partido (match)		departamento (state)		FARC (Colombian guerrilla)
Bogotá		Colombia		city (Manchester City)		sector		paz (peace)
Boyacá (Colombian state)		paz (peace)		madrid (Real Madrid)		tránsito (transit)		Maduro (Venezuela’s president)
Santa Marta (a city)		Maduro (Venezuela’s president)		Santa Fé (football team)		accidente (accident)		país (country)
Colombia		Colombia		junior (football team)		Bogotá		país (country)
gobierno (government)		gobierno (government)		gran (great)		Cundinamarca (Colombian State)		terroristas (terrorists)
nacional (national)		justicia (justice)		mejor (better)		cierre (closure)		colombianos (Colombians)
vehículos (vehicles)		colombianos (Colombians)		américa (football team)		total		pueblo (people)
personas (people)		Uribe (Colombia’s ex-president)		nacional (football team)		carril (lane)		justicia (justice)
#10 (Local Politics)	84	#8	76	#11 (Religion)	73	#2 (Dialogues)	65	#16 (Love)	60
Bogotá		más (more)		Dios (God)		más (more)		vida (lane)
Peñalosa (Bogotá’s mayor)		vida (life)		Señor (Lord)		ser (to be)		día (day)
Colombia		siempre (always)		amor (love)		vida (life)		amor (love)
Santos (Colombia’s president)		solo (only)		padre (father)		hijueputa (sonofabitch)		quiero (I want)
Petro (Bogotá’s ex-mayor)		mejor (better)		vida (life)		amor (love)		Dios (God)
FARC (Colombian guerrilla)		día (day)		gracias (thanks)		bien (good)		siempre (always)
paz (peace)		nunca (never)		corazón (heart)		solo (only)		corazón (heart)
Uribe (Colombia’s ex-president)		cosas (things)		Jesús (Jesus)		alguien (someone)		mejor (better)
gobierno (government)		gente (people)		misericordia (misericordy)		mamá (mom)		mierda (shit)
alcalde (mayor)		tiempo (time)		fuerza (strength)		hoy (today)		gracias (thanks)
#7 (Love)	57	#26	51	#23 (Love)	50	#15 (Love)	49	#9 (Portuguese and French)	49
más (more)		vida (life)		Dios (God)		quiero (life)		não (no)
vida (life)		mejor (better)		vida (life)		día (day)		est (is)
quiero (I want)		solo (only)		amor (love)		jajaja (laughter)		pas (not)
mierda (shit)		día (day)		siempre (always)		vida (life)		mais (more)
mejor (better)		años (years)		feliz (happy)		mejor (better)		quero (I want)
gente (people)		Colombia		corazón (heart)		hoy (today)		elle (he)
bien (good)		nunca (never)		gracias (thanks)		alguien (someone)		vou (you)
siempre (always)		siempre (always)		tiempo (time)		novio (boyfriend)		minha (my)
amor (love)		mundo (world)		nunca (never)		amo (I love)		hoje (today)
necesito (I need)		Bogotá		personas (people)		bien (good)		melhor (better)
#39 (Sports Football)	48	#5 (Love)	44	#31	43	#1	42	#29	42
partido (match)		amor (love)		días (days)		quiero (I want)		usted (formal you)
Santa fé (football team)		quiero (I want)		hoy (today)		jajaja (laughter)		quiero (I want)
gol (goal)		solo (only)		bueno (good)		usted (formal you)		vida (life)
hoy (today)		vida (life)		gente (people)		solo (only)		solo (only)
vamos (come on)		amo (I love)		vida (life)		mejor (better)		cosas (things)
bien (good)		dia (day)		ahora (now)		vida (life)		mejor (better)
#13	42	#38	41	#28	41	#27	40	#20	40
más (more)		más (more)		más (more)		jajaja (laughter)		amo (I love)
tan (so)		mejor (better)		vida (life)		más (more)		más (more)
vida (life)		quiero (I want)		tan (so)		fav		voy (I will)
hoy (today)		tan (so)		alguien (someone)		tan (so)		tan (so)
quiero (I want)		hoy (today)		quiero (I want)		quiero (I want)		quiero (I want)
solo (only)		mamá (mom)		solo (only)		vida (life)		vida (life)
#4	40	#35	40	#37	39	#14	39	#32	39
más (more)		más (more)		más (more)		más (more)		más (more)
vida (life)		tan (so)		usted (formal you)		quiero (I want)		ser (to be)
alguien (someone)		vida (life)		vida (life)		vida (life)		tan (so)
solo (only)		mierda (shit)		mejor (better)		tan (so)		voy (I will)
persona (person)		solo (only)		amor (love)		mejor (better)		solo (only)
ser (to be)		asi (like this/that)		Dios (God)		Dios (God)		vida (life)
#6	38	#0	37	#17	36	#18	36	#33	36
más (more)		más (more)		más (more)		más (more)		más (more)
vida (life)		quiero (I want)		quiero (I want)		tan (so)		amor (love)
quiero (I want)		tan (so)		tan (so)		mejor (better)		vida (life)
amor (love)		vida (life)		vida (life)		vida (life)		quiero (I want)
solo (only)		hoy (today)		mejor (better)		quiero (I want)		solo (only)
siempre (always)		día (day)		hoy (today)		jajaja (laughter)		tan (so)
#19	35	#36 (English)	34	#12	34	#3	33	#21	33
más (more)		like		más (more)		más (more)		más (more)
mejor (better)		love		hoy (today)		quiero (I want)		amor (love)
quiero (I want)		people		día (day)		siempre (always)		vida (life)
amor (love)		want		Dios (God)		vida (life)		solo (only)
vida (life)		get		vida (life)		voy (I will)		quiero (I want)
tan (so)		don’t		solo (only)		amor (love)		ser (to be)

Equations

$${\mathbf{w}}_1=(1,0,\dots ,0),\dots ,{\mathbf{w}}_V=(0,0,\dots ,1),$$

$$y_j:=P(w_j|w_i)=\frac{\exp (⟨{\text{upomega}}_j^{\prime },{\text{upomega}}_i⟩)}{{\sum }_{k=1}^V\exp (⟨{\text{upomega}}_k^{\prime },{\text{upomega}}_i⟩)},$$

$$E:=-\log P(w_j|w_i)=\log \sum _{k=1}^V\exp (⟨{\text{upomega}}_k^{\prime },{\text{upomega}}_i⟩)-⟨{\text{upomega}}_j^{\prime },{\text{upomega}}_i⟩.$$

$$P(d,w|\mathcal{C})=\{\begin{array}{ll}\frac{K}{1+K}\tilde{Q}(w)& \text{if }d=0,\\ \frac{1}{1+K}\tilde{P}(w|\mathcal{C})& \text{if }d=1.\end{array}$$

$$P(d=0|w,\mathcal{C})$$

$$P(d=1|w,\mathcal{C})$$

$$E=\sum _{k=1}^K\log \sigma (⟨{\text{upomega}}_k^{\text{n}},\mathbf{h}⟩)-\log \sigma (⟨\text{upomega},\mathbf{h}⟩),$$

$$D_r:=\frac{1}{2n_r}\sum _{{\mathbf{x}}_i,{\mathbf{x}}_j\in C_r}||{\mathbf{x}}_i-{\mathbf{x}}_j|{|}^2,$$

$$W_k:=\sum _{r=1}^k{D}_r.$$

$${\mathrm{Gap}}_B(k)=\frac{1}{B}\sum _{b=1}^B\log W_k^{(b)}-\log W_k,$$

$$s_k=\sqrt{1+\frac{1}{B}}\mathrm{sd}(k),$$

Full text

††journal: Cities

1 Introduction

Internet usage in Colombia, and in particular in Bogotá, has been increasing in the last years, not only because it constitutes the largest source of information of every kind, but also because of governmental efforts to make people from different social backgrounds recurrent Internet users [1], with the idea of reducing the digital gap in Colombia. For instance, from 2011 to 2014, several portions of the city have experienced an increase of up to 148.3% in homes with Internet connection [2].

By 2011, both Facebook and Twitter were the most popular social networks in Bogotá [1]. Moreover, 65.5% of Bogotá’s Internet users used social networks by 2014 [2]. This study concentrates on Twitter because it allows its users to write posts with a length ranging from 1 to 140 characters111Data was collected before the change in the maximum tweet length to 280 characters, making Twitter a tool for microblogging, a form of communication in which users express their opinion about several topics in short posts (tweets) [3].

By only taking the users’ posts and not the explicit relations between the users, our large data set is an interesting basis for testing unsupervised community detection methodologies. In fact, the only common factor that relates the users from our data set is that they are connected by geographical location. Such unsupervised community detection methodologies help to understand social phenomena that takes place in that geographical region and in a particular period of using data from any social network with no a priori knowledge of people’s relations.

With the objective of identifying the main topics treated by Bogotá’s population on Twitter, as well as detecting possible communities, we collected tweets emitted from Bogotá. Then, a Word2Vec model [4] was built in order to represent the set of tweets corresponding to each user as a vector in a vector space. The gap statistic [5] was used to estimate the number of clusters that could be formed using these vectors. Finally, a frequency distribution of words was built for each cluster, so that each cluster could be identified by its most frequent words, which ultimately characterizes a topic.

Our work acknowledges that online social networks constitute one of the main scenarios where people express their opinions, which makes them an outstanding source of information that allows the characterization of important topics for the citizenship. Therefore, we provide a robust method for studying cultural and political aspects of a society, which unveils groups of citizens that are actively concerned about particular topics that are relevant for the dynamics of a city. The main contribution of our work is the fully unsupervised and city-dependent method that we propose for discovering such groups. In turn, the identification of the main topics discussed by the citizens simplifies the task of targeting groups of people for promoting cultural, political or educational campaigns about very specific issues.

This paper is organized as follows. In Section 2 we present related work regarding methods for topic detection, models for text representation as well as a short review on community detection algorithms. Section 3 details material and methods. In Section 4 we present the theoretical aspects of the proposed method. In Section 5 the principal results are shown. We discuss these results in Section 6 with the purpose of providing some enlightenment on the way our method detected topics treated by Bogotá’s population. Finally, Section 7 presents the main conclusions of the paper and future work.

2 Related Work

In this section we review important milestones in the areas of topic detection, opinion-mining, text representation and community detection models, all of which are pertinent to our work.

Numerous methods have been created for topic detection in texts. The most successful and acknowledged ones are Latent Dirichlet Allocation (LDA) [6], Latent Semantic Analysis (LSA) (or Latent Semantic Indexing (LSI)) [7], and Correlated Topic Models (CTM) [8]. However, people always tune and mix these methods in larger pipelines that allow them to study a singular application in which they are interested. In particular, analyzing posts from Twitter is a challenging task because these are short texts that lack of context. Moreover, many of these short texts are not words of an official language, both because of misspelling and because of the usage of Internet slang. However, several studies have tried to adapt to these inconveniences. A model [9] based on TwitterLDA [10] (which is an adaptation of LDA to short texts such as tweets) was constructed to take advantage of Twitter as a source of real world event information. This model aimed to detect emerging events that could affect a geographical region in a city, and also quantitatively estimated the impact of such an event on the population nearby the event location. Another study [11] proposed a model to detect related topics based on clustering performed over a TF-IDF representation of tweets. The clustering was done by including weights between words called Associative Gravity Force [12] (along with other measures of similarity between words, as well as ranking of words), that accounted for the frequency of pairs of words occurring together. Also, graph-based approaches have been recently proposed [13], in which hash-tags are used to identify communities and topics by finding frequent co-occurring hash-tags.

Other non common methods have also been used to study texts from Twitter, like [14], in which Formal Concept Analysis (FCA) [15, 16], a mathematical application of lattices and ordered sets to the process of concept formation, was used as an alternative approach to topic detection. Authors use FCA because it deals with several problems that the traditional methods suffer such as the unknown number of topics, the difficulty of these methods to adapt to new topics, among others.

It is worth noting that the majority of work has been done using English text corpora with valuable results [17]. However, Spanish is significantly a more inflected language than English, and this difference could pose problems. For example, supervised machine learning methods for the topic classification of annotated Spanish tweets modeled with $n$-grams have shown to be insufficient [18]. Better attempts to deal with the several research branches of topic detection and opinion-mining in Spanish have taken place. For instance, in the field of opinion mining, studies like [19] proposed a lexicon-based model that adapts to specific domains in Spanish for polarity classification of film reviews. Also, polarity classification in Spanish tweets has been treated in [20], where hybrid systems that bring together knowledge from lexical, syntactic and semantic structures in the Spanish language, as well as machine learning techniques used with the bag-of-words representation, have shown improvements over the sole bag of words approach. Besides, the clever creation of a corpus called MeSiento (Spanish for “I feel”) [21] allowed a robust unsupervised method for polarity classification of Spanish tweets that reached accuracy levels close to the ones obtained with supervised algorithms.

Nonetheless, these methods applied to sentiment analysis tasks depend a lot on annotated dictionaries which do not contain Bogotá’s jargon. As a matter of fact, few attempts have been made to make small topic-specific dictionaries such as [22], where the political sentiment towards Bogotá mayoral candidates for 2015 was analyzed using Twitter and a political sentiment dictionary defined in the Colombian political context. A second study briefly examined the sentiment of tweets from Bogotá with words related with health symptoms [23]. A third study examined the results of 2015 Colombian regional elections and compared them with political ideology and Twitter activity of the candidates [24].

In the end, it is clear that an unsupervised model for text representation is needed to give robust topic-independent text representations. One of the most successful and widely used text representation models is Word2Vec, which has proven to give good results regardless of the language in opinion mining and topic detection duties. For instance, [25] combined Word2Vec and a bag-of-words document classifier, and showed that Word2Vec provided word embeddings that produced more stable results when doing cross-domain classification experiments. Also, since Word2Vec was first introduced, there has been some research trying to improve and fine-tune word embeddings. Such is the case of [26] that proposes a hybrid model between skip-gram model and continuous bag of words (CBOW) called mixed word embedding (MWE). All in all, we choose word embedding models such as Word2Vec for being able to embed semantic similarities between words in a similarity metric defined over a Euclidean vector space.

Advances in sentiment analysis have been reported in recent works such as [17], where state of the art methods were surveyed and compared. Deep learning methods are being used in works such as [27] to analyze sentiments in Persian texts. Deep convolutional neural networks have been also investigated to analyze sentiments in Twitter [28]. Deep learning based methods have been used to detect malicious accounts in location-based social networks [29]. One recent work used a Bayesian network and fuzzy recurrent neural networks for detecting subjectivity [30].

With regard to one of the particular objectives of our work: detecting communities, several methods have been developed in the last couple of decades to solve the so-called planted $\ell$-partition model, where the structure of graphs are studied to find densely connected groups of nodes (see refs [31, 32] for excellent reviews). More modern methods based on embedding communities in low-dimensional vector spaces try to solve problems such as node clustering, node classification, low-dimensional visualizations, edges prediction, among others with great success [33, 34]. However, we shall point out that this is a very active area of research with many facets, and as argued in [35], community detection should not be considered as a well-defined problem, but instead, should be motivated by particular reasons. In this sense, our motivation for detecting communities is to find groups of people with a clear topic of interest, regardless of whether such groups of people follow each other on Twitter. This means that we do not know from the beginning any connection between the nodes (users), and we aim to detect communities solely based on the data that characterizes each node, i.e. the text representation of each user’s tweets.

The contribution of this paper is twofold: first, we proposed a method to automatically identify digital communities of a city grouped by topic of interest, and second, we collected a set of tweets of Bogotá’s citizens to illustrate the proposed method.

3 Material and methods

An overview of the proposed method is depicted in Figure 1, which is inspired by the ideas found in  [36]. We first crawl a set of tweets, which are stored in a document data base. Then, we generate vector representations of texts using Word2Vec. We selected this model because it has been the seed for all word embedding models, and it is the most widely used model, despite the existence of newer and very successful word embedding models such as fastText [37, 38, 39], BERT [40], Swivel [41] and ELMo [42]. Afterwards, clustering analysis is performed in the embedded space to find latent topics. Each user is projected in a 2D visualization in which the obtained latent topics are colored. It is of the uttermost importance to notice that we do not create a graph with explicit edges between the users, but rather let the latent topics found in each user’s tweets to create implicit edges. The following sections describe in detail each of these components.

3.1 Crawler

A crawler component was implemented in Java using the Twitter Streaming API, which allowed to collect a set of 2,476,426 tweets of Bogotá in a period of 111 days (from August 2015 to December of 2015). The query only searched for tweets that matched with the string “Bogotá” in the field “place” of the tweet meta-data.

3.2 Tweets data set

Tweets corresponding to the same user form a document. All the documents form the corpus. The set of words composing the corpus is the vocabulary. The distribution of tweet and document length of the corpus is shown in Figure 2. It is worth noting that most of the tweets have 20-60 characters. The average length in characters of the tweets is 55 and the average length in tokens of the documents is 639.

3.3 Pre-processing

We decided to discard tweets with less than 20 characters because very short texts usually lack of significant information. From the tweets with 20 or more characters, a total of 58,644 documents were created. The documents were tokenized using the NLTK’s Tweet Tokenizer [43], which allowed to preserve emoticon-like (and smileys) characters in tweets. The tokens were not stemmed because Word2Vec deals with different conjugations of words.

3.4 Word2Vec embedding

With these documents, a Word2Vec model with a context distance of 6 words was trained. Words that appeared at least 10 times in the corpus were selected from the vocabulary to train the model. This subset of the corpus vocabulary (or model’s vocabulary) was composed of 55,168 words. Word2Vec represents each document (the average of the embedded vector words composing the document) as a vector in a vector space whose cardinality was set to 150 dimensions.

3.5 Documents database

Having the vector representation in the 150 dimensional space, we discarded the ones that contained less than 40 word occurrences from the model’s vocabulary. This was done with the purpose of examining documents that represented active Twitter users. A total of 30,746 documents satisfied this condition.

3.6 Clustering of documents (users)

In order to identify the main topics treated by Bogotá’s Twitter users we used the k-means algorithm (with the Python’s scikit-learn module [44]) on the vector representation of the documents. To determine the number of clusters $k$ in the cluster analysis, a gap statistic study was performed.

3.7 Frequency distribution of words within clusters

Once $k$ was estimated, we took the 15 most representative documents of each cluster to build a frequency distribution of words that allowed us to easily identify the topics represented in each cluster. Moreover, a tweet length distribution of the 15 documents of each cluster was also built in order to recognize which topics demanded users to write longer or shorter tweets.

4 Theoretical framework

This section presents the theoretical framework used in each component of the proposed method.

4.1 Word2Vec Embedding

Word embeddings constitute a solution to the task of numerically representing words as semantics and syntax carriers within sentences. Let $\mathcal{V}=\{w_{i}:i=1,2,\ldots,V\}$ be the ordered set of words, or vocabulary, contained in the corpus, where $w_{i}$ is the $i$-th word, and $V$ is the size of the vocabulary. Generally, words are represented as hot-vectors, which are the $V$-dimensional canonical basis vectors,

[TABLE]

where $\boldsymbol{w}_{i}$ is the hot-vector representation of the word $w_{i}$. Clearly $\langle\boldsymbol{w}_{i},\boldsymbol{w}_{j}\rangle:=\sum_{k}w_{ik}w_{jk}=\delta_{ij}$, where $\delta_{ij}$ is the Kronecker’s delta and $w_{ik}$ is the $k$-th component of the $i$-th word hot-vector, which shows that in this canonical basis of $\mathbb{R}^{V}$ there cannot be a relation between words, since they are pairwise orthonormal. Therefore, Mikolov et al. [45] proposed a model to build a projector $\boldsymbol{\mathcal{W}}:\mathbb{R}^{V}\to\mathbb{R}^{N}$, where $N\ll V$, that maps hot-vectors into embedded $N$-dimensional vectors. That is, $\boldsymbol{\mathcal{W}}(\boldsymbol{w}_{i})\mapsto\boldsymbol{\upomega}_{i}$, where $\boldsymbol{\upomega}_{i}$ is the embedded $N$-dimensional representation of the word $w_{i}$. The way of constructing $\boldsymbol{\mathcal{W}}$ is via a multi-layered neural network that can be arranged in two different ways. The first way results in the Continuous Bag of Words (CBOW) model, and the second way results in the skip-gram model. This work centers in the CBOW model, in which the neural network has the job of predicting a target word given a set of words called context words. The context words of a word are defined as the set of words that are at a distance less than or equal to $c$ from each occurrence of the word in the corpus, where $c$ is some integer that one defines. For instance, if the corpus contains food reviews, one would expect the neural network to predict “food” when the context words are $\{$“delicious”, “yummy”, “exquisite”$\}$, and do not predict “cat”. The neural network can be depicted as in Figure 3, where the input layer consists of $C\times(V\times 1)$ neurons ($C$ is the number of words in the set of context words), the hidden layer of $N$ neurons, and the output layer of $V$ neurons.

To see how $\boldsymbol{\mathcal{W}}$ is constructed, consider the simplified problem of a one word context. In this case, the input layer receives a word hot-vector $\boldsymbol{w}_{i}$, and acts linearly on it with a $N\times V$ matrix $\boldsymbol{W}$, whose components $W_{ji}$ are the weights between the $i$-th neuron of the input layer, and the $j$-th neuron of the hidden layer. This layer can be represented by an $N$-dimensional vector $\boldsymbol{h}$ whose components are the sum of the weights received by each neuron, i.e. $h_{k}=(\boldsymbol{W}\boldsymbol{w}_{i})_{k}=W_{ki}$. Here, $\boldsymbol{W}$ can be identified as the matrix form of the projector $\boldsymbol{\mathcal{W}}$, where the $i$-th column of $\boldsymbol{W}$ is $\boldsymbol{\upomega}_{i}$. In a similar fashion, $\boldsymbol{W}^{\prime}$ is a $V\times N$ weight matrix between the hidden and the output layer. Note that both the $i$-th column of $\boldsymbol{W}$ and the $i$-th row of $\boldsymbol{W}^{\prime}$ are $N$-dimensional vector representations of the word $w_{i}$. Moreover, since $h_{k}=W_{ki}$, then $\boldsymbol{h}=\boldsymbol{\upomega}_{i}$. This allows $\langle\boldsymbol{\upomega}^{\prime}_{j},\boldsymbol{h}\rangle=\langle\boldsymbol{\upomega}^{\prime}_{j},\boldsymbol{\upomega}_{i}\rangle$ to be a score of similarity between the different vector representations of the words $w_{i}$ and $w_{j}$. These scores allow the definition of a softmax multinomial distribution,

[TABLE]

where $y_{j}$ is the output of the $j$-th neuron in the output layer. Eq. (2) is the estimated probability of $w_{i}$ being a context word of $w_{j}$. To learn the correct probabilities, a loss function $E$ is defined as

[TABLE]

Since the goal is to minimize $E$, then $\langle\boldsymbol{\upomega}^{\prime}_{j},\boldsymbol{\upomega}_{i}\rangle$ must become larger in the learning process, meaning that words that share a context must have similar vector representations, and words that do not share contexts must have dissimilar vector representations. Notice that computing $E$ is time-consuming, since normally $V$ is a big number ($\sim 10^{4}$).

Furthermore, when $C>1$, $\boldsymbol{h}$ is defined as the average vector of the context words’ vector representations. Here the loss function is $E:=-\log P(w_{j}|\mathcal{C})$, where $\mathcal{C}$ is a set of $C$ words chosen from $\mathcal{V}$. This probability can be expressed as in Eq. (2) by changing $\boldsymbol{\upomega}_{i}$ by $\boldsymbol{h}$. Notice that there are $\binom{V}{C}$ ways of picking contexts, and the number of calculations increase dramatically. To reduce this number, negative sampling [45] is used. Let $\tilde{P}(w)$ be the term frequency distribution of words. Let $\tilde{Q}(w)$ be a noise distribution built by taking the probability $P(w)$ of each word and raising them to the power of 3/4 (this allows less frequent words to have more probability of being drawn from $\tilde{Q}$ [46]), and then renormalizing. The problem of minimizing $E$ is converted into a binary classification problem as follows. Given a context $\mathcal{C}$, we pick a word $w$ from $\mathcal{V}$ such that $\mathcal{C}$ are context words for $w$. This pair $(w,\mathcal{C})$ is called a true sample and is labeled by $d=1$. From the noise distribution $\tilde{Q}$, $K$ words $\{w^{\text{n}}_{1},\ldots,w^{\text{n}}_{K}\}$ are drawn. The pairs $(w^{\text{n}}_{k},\mathcal{C})$ are negative samples and are labeled by $d=0$. Therefore, the binary classification problem can be stated as maximizing the joint probability of $(d,w)$, where $w$ can refer to a word both from the true sample or from the negative samples [47]:

[TABLE]

Since the actual empirical distribution $\tilde{P}(w|\mathcal{C})$ is not known, the one defined in Eq. (2) is used. Now, to reduce the calculations, under some approximations [47], it is possible to write Eq. (4) as two simple equations:

[TABLE]

Note that these are sigmoid functions $\sigma$, of the same argument, except for a sign. From Eq. (5) it can be seen that if $\boldsymbol{\upomega}$ and $\boldsymbol{h}$ are dissimilar (similar) then the mentioned probability will be close to 1 (0). Similarly, in Eq. (6), if $\boldsymbol{\upomega}$ and $\boldsymbol{h}$ are similar (dissimilar), the probability will also be close to 1 (0). In this case, it can be shown that the error function takes the form

[TABLE]

for a target word $w$. To see negative sampling in the context of the skip-gram model (also used instead of CBOW), see [46].

4.2 Gap statistic

The gap statistic allows the estimation of the number of clusters in a data set [5]. Consider a data set $\{\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{N}\}$ of $N$ samples and $M$ features. Let $C_{r}$ be the $r$-th cluster found with K-Means out of $k$ clusters (other clustering algorithms can be implemented), containing $n_{r}=|C_{r}|$ samples. The within-cluster dispersion is defined as

[TABLE]

which is similar to the variance except for a factor of $1/n_{r}$. The total within-cluster dispersion for $k$ clusters is

[TABLE]

$W_{k}$ is normally used to estimate the “correct” number of clusters $K$ via the elbow method. On the other hand, the gap statistic uses null reference distributions. These are each constructed by finding the range in the $M$-dimensional space of the samples and generating $N$ data points with a uniform distribution. Next, $k$ clusters are computed for each reference distribution. If there are $B$ such distributions, then the gap statistic is defined as

[TABLE]

where $W_{k}^{(b)}$ is the total within-cluster dispersion of the $b$-th reference distribution for $k$ clusters. It can be argued [5] that $\operatorname{Gap}_{B}(k)$ reaches a maximum for $k=K$ when the cluster centroids are aligned in an equally spaced fashion. Also, the uncertainty of $\operatorname{Gap}_{B}(k)$ can be estimated to be

[TABLE]

where $\operatorname{sd}(k)$ is the standard deviation of the logarithm of the $B$ reference distributions’ total within-cluster dispersion. Thus, with a 1-sigma certainty, $K$ is the value of $k$ for which $\operatorname{Gap}_{B}(k)\pm s_{k}$ reaches its maximum.

5 Results

Once the Word2Vec model was trained, the documents were represented by the average of its Word2Vec representations. The gap statistic was computed for several quantity of clusters $k=5,10,\ldots,140$, resulting in Figure 4. The curve shown in this figure resembles a $\log$ curve. This happens because the Word2Vec model uses all the dimensions to represent the similarity relations between word vectors. Since texts are normally rich in words, it is expected that no clear clusters are formed, and therefore the number of clusters is quite indistinguishable. This is also supported by the fact that Twitter users might tweet about different topics, making their document’s vector to be assigned to clusters that group 2 or more topics. Despite these inconveniences, the gap statistic allows us to estimate the number of clusters. From the figure, it can be seen that the curve begins to flatten down, or to form an elbow, around $k=40$.

The result of K-means clustering with 40 clusters is shown in Figure 5, where the 15 most representative documents of each cluster were plotted, using PCA (Figure 5a) and t-SNE (Figure 5b), using different colors for each cluster.

We established the 15 most representative documents of each cluster by sorting in ascending order the euclidean distance between each document of the cluster and its respective cluster centroid. With each set of 15 documents, a word frequency distribution was built in order to get the most frequent words of each cluster. A tweet length distribution was also built with each of these sets. The most frequent words in each cluster tell us if the topics can be easily defined. From both visualizations it is seen that there are two types of clusters: the ones corresponding to a mega-cluster, and the ones that are certainly different from the rest of the corpus, called one-topic clusters because their topic can be easily defined. The one-topic clusters are tagged in the figures with their respective topic (except from “Love” and “Religion”, which can be found within the mega-cluster). It can be seen that PCA makes a sparse visualization of the documents corresponding to one-topic clusters, while t-SNE groups them and plots them apart from the mega-cluster. It should be noted that the one-topic clusters can be used to treat documents as nodes in a graph, whose edges connect nodes from the same one-topic clusters. This graph could then be used by a semi-supervised community detection method such as [48] in order to label the remaining nodes from the mega-cluster.

The most frequent words for each cluster are presented in Table 1. It is noteworthy that even though clusters 25 and 30 are related to the same topic, documents in cluster 30 are more news-like written, while 25 contained documents full of personal opinions about Country Politics. The same relation occurs between clusters 34 and 39. Cluster 34 is full of documents with well-written tweets that try to inform the situation of a particular football team, or match, whereas cluster 39 contains documents that express feelings about specific teams or football events. It is important to point out that cluster 2 was filled with dialogue-like written tweets, i.e. tweets that have conversations in them. In the case of clusters 22 and 24, corresponding to news, it can be identified that the combination of words used in these types of reporting tweets is very peculiar of those clusters. This is clear because even though the topic news covers plenty of topics, the way the tweets are written is perceived by Word2Vec.

6 Discussion

Remarkably, many clusters contained documents that constantly make reference to love. The similarity between these clusters can be seen because they share many common words. Also, cluster 9 is a very particular one because it encloses documents containing tweets both written in French and Portuguese.

From the average tweet length of each cluster, it can be easily seen that longer tweets generally are part of a document contained in a cluster that represents a specific topic. On the other hand, clusters with short tweet length average consist of documents with tweets that express personal experiences. From the table, it is seen that there are plenty of clusters with these sorts of documents, indicating that people tend to express their personal experiences with the same set of words and in very similar semantic expressions.

To make Table 1 easier to read, a PCA visualization of the cluster centroids is shown in Figure 6. In this figure, the clusters with the longer tweet length average appear to be away from the mega-cluster, comprised of all the short tweet length average clusters. This phenomenon allows us to propose that people tend to share their personal experiences in shorter tweets, while they give opinions of community important topics in longer texts.

Overlapping of document vectors in two topics means that these users belong to both communities, which is normal because typically a user is interested in more than one topic. This is known in machine learning as soft clustering, in which an object can belong to more than one cluster. In our approach this analysis was not conducted, but it would be interesting to address this aspect in future work.

6.1 Threats to validity

It is worth noting that our approach is not using other information available in Twitter such as user account meta-data, re-tweets, likes, URLs, images, location, geo-referenced data (when available), etc. We concentrate our analysis only in the text content. However, these complementary information could add value to better represent a user and at the same time to modify the obtained communities. Other issue in our approach is the amount of tweets used in the experimentation. We gathered data from Bogota users in a period of six months, but It is not clear what is considered enough data. However, gathering more data is a time consuming task and high demanding disk space. We considered that this issue can be addressed in future work to analyze the impact of the detected communities in a larger period of time.

6.2 Policy implications

It is important to remark that our work can be used by government and private entities to develop cultural, political or educational campaigns, which are the most valuable intangible fields in a society. We believe that the method proposed in this paper is especially useful in developing countries because of the gap or disconnection that exists between politicians in the government and the citizenship, because our method easily identifies the main concerns of a society. As an example, during the time we gathered the tweets, every city in Colombia was going through one of the deepest changes of the country because of the peace treaty between the Colombian government and the FARC guerrilla, which was discovered by our method as one of the main topics discussed in Bogotá. This early information could have been used by the government to explain the treaty specifically to those confused or uninformed.

7 Conclusion and future work

This paper presented a method to automatically identify communities using citizens data from the social network Twitter. Up to knowledge, this is the first study that analyzes Twitter data from Bogota to automatically detect communities. The use of machine learning methods such as neural networks and dimensionality reduction algorithms to detect communities is the base of the proposed method. Results show that this method can find out groups of citizens that share common topics such as politics, news, religion, sports, languages, among others. This is an interesting tool that could be used by the local government to support making decision processes in which what communities express can provide valuable information.

As future work we want to compare our approach with the communities that can be obtained when data is modeled as a graph, which will be challenging because the obtained communities will not be necessarily comparable. We want to explore in the future the use of other data available in Twitter such as meta-data user account, re-tweets, likes, URLs, etc. It is possible that complementary information of users generates value to the concept of community. We also want to gather more Tweets to increase the sample data an analyze how the communities change in the time. It would be interesting to analyze how communities are influenced by social phenomena, events and period of the year.