Learning to Decipher Hate Symbols

TL;DR

This paper introduces a new task of deciphering hate symbols in social media, leveraging neural network models and a novel Twitter corpus to improve understanding of hidden hate speech connotations.

Contribution

It proposes a new deciphering task, creates a symbol-rich Twitter dataset, and develops neural models including a Variational Decipher for better generalization.

Findings

Sequence-to-Sequence models can decode hate symbols based on context.

The Variational Decipher outperforms other models on unseen hate symbols.

The approach enhances understanding of covert hate speech in social media.

Abstract

Existing computational models to understand hate speech typically frame the problem as a simple classification task, bypassing the understanding of hate symbols (e.g., 14 words, kigy) and their secret connotations. In this paper, we propose a novel task of deciphering hate symbols. To do this, we leverage the Urban Dictionary and collected a new, symbol-rich Twitter corpus of hate speech. We investigate neural network latent context models for deciphering hate symbols. More specifically, we study Sequence-to-Sequence models and show how they are able to crack the ciphers based on context. Furthermore, we propose a novel Variational Decipher and show how it can generalize better to unseen hate symbols in a more challenging testing setting.

Tables

Table 1. Table 1: The BLEU, ROUGE-L and METEOR scores on testing datasets. VD refers to the Variational Decipher. D 𝐷 D is the entire testing dataset. D s subscript 𝐷 𝑠 D_{s} is the first part of D 𝐷 D and D d subscript 𝐷 𝑑 D_{d} is the second part. The better results are in bold.

Dataset	Method	BLEU	ROUGE-L	METEOR
$D_s$	Seq2Seq	37.80	41.05	36.67
	VD	34.77	32.96	31.03
$D_d$	Seq2Seq	25.44	12.96	5.54
	VD	28.38	14.01	5.41
$D$	Seq2Seq	33.96	32.32	26.98
	VD	32.75	27.00	23.16

Table 2. Table 2: The results of human evaluation on two separate testing datasets D s subscript 𝐷 𝑠 D_{s} and D d subscript 𝐷 𝑑 D_{d} .

Dataset	Seq2Seq Lose	Seq2Seq Win	Tie
$D_s$	31.0%	32.0%	37.0%
$D_d$	30.5%	22.0%	47.5%

Equations

$$Obj=\sum _{(u,s,d^{\ast })\in X}\log p(d^{\ast }|(u,s))$$

$$c_u,h_u=f_u(u)$$

$$c_{sw},h_{sw}=f_{sw}(s_w)$$

$$c_{sc},h_{sc}=f_{sc}(s_c)$$

$$h=h_{sw}\oplus h_{sc}$$

$$w_t=\sigma (l_w(d_{t-1}\oplus e_{t-1}))$$

$$a_t=\sum _{i=1}^T{w}_{ti}{h}_i$$

$$b_t=\sigma (l_c(d_{t-1}\oplus a_t))$$

$$o_t,e_t=k(c_u\oplus b_t,e_{t-1})$$

$$p(d_t|u,s)=\sigma (l_o(o_t))$$

$$\begin{array}{rl}Obj& =\sum _{(u,s,d^{\ast })\in X}\log p(d^{\ast }|(u,s))\\ & =\sum _{(u,s,d^{\ast })\in X}\log {\int }_{z}p(d^{\ast }|z)p(z|(u,s))dz\end{array}$$

$$Obj=$$

$$D_{KL}[p_{\alpha }(z|d^{\ast },u,s)||p_{\beta }(z|u,s)]$$

$$x,h_d=f_d(d^{\ast })$$

$$c=c_u\oplus c_{sw}\oplus c_{sc}$$

$$\mu ,\Sigma =g(x\oplus c)$$

$${\mu }^{\prime },{\Sigma }^{\prime }=g^{\prime }(c)$$

$$o_t,e_t=k(z\oplus c\oplus b_t,e_{t-1})$$

$$\mathcal{L}=$$

$$=$$

$$D_{KL}[p_{\alpha }(z|d^{\ast },u,s)||p_{\beta }(z|u,s)]$$

Full text

1 Introduction

The statistics are sobering. The Federal Bureau of Investigation of United States111https://www.fbi.gov/news/stories/2016-hate-crime-statistics reported over 6,000 criminal incidents motivated by bias against race, ethnicity, ancestry, religion, sexual orientation, disability, gender, and gender identity during 2016. The most recent 2016 report shows an alarming 4.6% increase, compared with 2015 data222https://www.fbi.gov/news/stories/2015-hate-crime-statistics-released. In addition to these reported cases, thousands of Internet users, including celebrities, are forced out of social media due to abuse, hate speech, cyberbullying, and online threats. While such social media data is abundantly available, the broad question we are asking is—What can machine learning and natural language processing do to help and prevent online hate speech?

The vast quantity of hate speech on social media can be analyzed to study online abuse. In recent years, there has been a growing trend of developing computational models of hate speech. However, the majority of the prior studies focus solely on modeling hate speech as a binary or multiclass classification task Djuric et al. (2015); Waseem and Hovy (2016); Burnap and Williams (2016); Wulczyn et al. (2017); Pavlopoulos et al. (2017).

While developing new features for hate speech detection certainly has merits, we believe that understanding hate speech requires us to design computational models that can decipher hate symbols that are commonly used by hate groups. Figure 1 shows an example usage of hate symbols in an otherwise seemingly harmless tweet that promotes hate. For example, Aryan Warrior is a longstanding racist prison gang based in the Nevada prison system. WPWW is the acronym for White Pride World Wide. The hate symbols 1488 and 2316 are more implicit. 14 symbolizes the 14 words: “WE MUST SECURE THE EXISTENCE OF OUR PEOPLE AND A FUTURE FOR WHITE CHILDREN”, spoken by members of the Order neo-Nazi movement. H is the 8th letter of the alphabet, so 88=HH=Heil Hitler. Similarly, W is the 23rd and P is the 16th letter of the alphabet, so 2316=WP=White Power.

In this work, we propose the first models for deciphering hate symbols. We investigate two families of neural network approaches: the Sequence-to-Sequence models Sutskever et al. (2014); Cho et al. (2014) and a novel Variational Decipher based on the Conditional Variational Autoencoders Kingma and Welling (2014); Sohn et al. (2015); Larsen et al. (2016). We show how these neural network models are able to guess the meaning of hate symbols based on context embeddings and even generalize to unseen hate symbols during testing. Our contributions are three-fold:

• •

We propose a novel task of learning to decipher hate symbols, which moves beyond the standard formulation of hate speech classification settings.

• •

We introduce a new, symbol-rich tweet dataset for developing computational models of hate speech analysis, leveraging the Urban Dictionary.

• •

We investigate a sequence-to-sequence neural network model and show how it is able to encode context and crack the hate symbols. We also introduce a novel Variational Decipher, which generalizes better in a more challenging setting.

In the next section, we outline related work in text normalization, machine translation, conditional variational autoencoders, and hate speech analysis. In Section 3, we introduce our new dataset for deciphering hate speech. Next, in Section 4, we describe the design of two neural network models for the decipherment problem. Quantitative and qualitative experimental results are presented in Section 5. Finally, we conclude in Section 6.

2 Related Work

2.1 Text Normalization in Social Media

The proposed task is related to text normalization focusing on the problems presented by user-generated content in online sources, such as misspelling, rapidly changing out-of-vocabulary slang, short-forms and acronyms, punctuation errors or omissions, etc. These problems usually appear as out-of-vocabulary words. Extensive research has focused on this task Beaufort et al. (2010); Liu et al. (2011); Gouws et al. (2011); Han and Baldwin (2011); Han et al. (2012); Liu et al. (2012); Chrupała (2014). However, our task is different from the general text normalization in social media in that instead of the out-of-vocabulary words, we focus on the symbols conveying hateful meaning. These hate symbols can go beyond lexical variants of the vocabulary and thus are more challenging to understand.

2.2 Machine Translation

An extensive body of work has been dedicated to machine translation. Knight et al. (2006) study a number of natural language decipherment problems using unsupervised learning. Ravi and Knight (2011) further frame the task of machine translation as decipherment and tackle it without parallel training data. Machine translation using deep learning (Neural Machine Translation) has been proposed in recent years. Sutskever et al. (2014) and Cho et al. (2014) use Sequence to Sequence (Seq2Seq) learning with Recurrent Neural Networks (RNN). Bahdanau et al. (2015) further improve translation performance using the attention mechanism. Google’s Neural Machine Translation System (GNMT) employs a deep attentional LSTM network with residual connections Wu et al. (2016). Recently, machine translation techniques have been also applied to explain non-standard English expressions Ni and Wang (2017). However, our deciphering task is not the same as machine translation in that hate symbols are short and cannot be modeled as language.

Our task is more closely related to  Hill et al. (2016) and  Noraset et al. (2017).  Hill et al. (2016) propose using neural language embedding models to map the dictionary definitions to the word representations, which is the inverse of our task.  Noraset et al. (2017) propose the definition modeling task. However, in their task, for each word to be defined, its pre-trained word embedding is required as an input, which is actually the prior knowledge of the words. However, such kind of prior knowledge is not available in our decipherment task. Therefore, our task is more challenging and is not simply a definition modeling task.

2.3 Conditional Variational Autoencoder

Unlike the original Seq2Seq model that directly encodes the input into a latent space, the Variational Autoencoder (VAE) Kingma and Welling (2014) approximates the underlying probability distribution of data. VAE has shown promise in multiple generation tasks, such as handwritten digits Kingma and Welling (2014); Salimans et al. (2015), faces Kingma and Welling (2014); Rezende et al. (2014), and machine translation Zhang et al. (2016). Conditional Variational Autoencoder Larsen et al. (2016); Sohn et al. (2015) extends the original VAE framework by incorporating conditions during generation. In addition to image generation, CVAE has been successfully applied to some NLP tasks. For example, Zhao et al. (2017) apply CVAE to dialog generation, while Guu et al. (2018) use CVAE for sentence generation.

2.4 Hate Speech Analysis

Closely related to our work are Pavlopoulos et al. (2017); Gao et al. (2017). Pavlopoulos et al. (2017) build an RNN supplemented by an attention mechanism that outperforms the previous state of the art system in user comment moderation Wulczyn et al. (2017). Gao et al. (2017) propose a weakly-supervised approach that jointly trains a slur learner and a hate speech classifier. While their work contributes to the automation of harmful content detection and the highlighting of suspicious words, our work builds upon these contributions by providing a learning mechanism that deciphers suspicious hate symbols used by communities of hate to bypass automated content moderation systems.

3 Dataset

In this section, we describe the dataset we collected for hate symbol decipherment.

3.1 Hate Symbols

We first collect hate symbols and the corresponding definitions from the Urban Dictionary. Each term with one of the following hashtags: #hate, #racism, #racist, #sexism, #sexist, #nazi is selected as a candidate and added to the set $S_{0}$. We collected a total of 1,590 terms. Next, we expand this set by different surface forms using the Urban Dictionary API. For each term $s_{i}$ in set $S_{0}$, we obtain a set of terms $R_{i}$ that have the same meaning as $s_{i}$ but with different surface forms. For example, for the term brown shirt, there are four terms with different surface forms: brown shirt, brown shirts, Brownshirts, brownshirt. Each term in $R_{i}$ has its own definition in Urban Dictionary, but since these terms have exactly the same meaning, we select a definition $d_{i}$ with maximum upvote/downvote ratio for all the terms in $R_{i}$. For example, for each term in the set $R_{i}$={brown shirt, brown shirts, Brownshirts, brownshirt}, the corresponding definition is “Soldiers in Hitler’s storm trooper army, SA during the Nazi regime…” After expanding, we obtain 2,105 distinct hate symbol terms and their corresponding definitions. On average, each symbol consists of 9.9 characters, 1.5 words. Each definition consists of 96.8 characters, 17.0 words.

3.2 Tweet Collection

For each of the hate symbols, we collect all tweets from 2011-01-01 to 2017-12-31 that contain exactly the same surface form of hate symbol in the text. Since we only focus on hate speech, we train an SVM  Cortes and Vapnik (1995) classifier to filter the collected tweets. The SVM model is trained on the dataset published by  Waseem and Hovy (2016). Their original dataset contains three labels: Sexism, Racism, and None. Since the SVM model is used to filter the non-hate speech, we merge the instances labeled as sexism and racism, then train the SVM model to do binary classification. After filtering out all the tweets classified as non-hate, our final dataset consists of 18,667 (tweet, hate symbol, definition) tuples.

4 Our Approach

We formulate hate symbol deciphering as the following equation:

[TABLE]

$X$ is the dataset, $(u,s,d^{*})$ is the (tweet, symbol, definition) tuple in the dataset. The inputs are the tweet and the hate symbol in this tweet. The output is the definition of the symbol. Our objective is to maximize the probability of the definition given the (tweet, symbol) pair. This objective function is very similar to that of machine translation. So we first try to tackle it based on the Sequence-to-Sequence model, which is commonly used in machine translation.

4.1 Sequence-to-Sequence Model

We implement an RNN Encoder-Decoder model with attention mechanism based on Bahdanau et al. (2015). We use GRU  Cho et al. (2014) for decoding. However, instead of also using GRU for encoding, we found that LSTM  Hochreiter and Schmidhuber (1997) performs better on our task. Therefore, our Seq2Seq model uses LSTM encoders and GRU decoders. An overview of our Seq2Seq model is shown in Figure 2. The computation process is shown as the following equations:

[TABLE]

$u$ is the word embedding of the tweet text, $s_{w}$ is the word embedding of the hate symbol, $s_{c}$ is the character embedding of the symbol. $f_{u}$, $f_{sw}$, and $f_{sc}$ are LSTM functions. $c_{u}$, $c_{sw}$, $c_{sc}$ are the outputs of the LSTMs at the last time step and $h_{u}$, $h_{sw}$, $h_{sc}$ are the hidden states of the LSTMs at all time steps. We use two RNN encoders to encode the symbol, one encodes at the word level and the other one encodes at the character level. The character-level encoded hate symbol is used to provide the feature of the surface form of the hate symbol while the word-level encoded hate symbol is used to provide the semantic information of the hate symbol. The hidden states of the two RNN encoders for hate symbols are concatenated:

[TABLE]

$c_{u}$ is the vector of encoded tweet text. The tweet text is the context of the hate symbol, which provides additional information during decoding. Therefore, the encoded tweet text it is also fed into the RNN decoder. The detailed attention mechanism and decoding process at time step $t$ are as follows:

[TABLE]

$w_{t}$ is the attention weights at time step $t$ and $w_{ti}$ is the $i_{th}$ weight of $w_{t}$. $d_{t-1}$ is the generated word at last time step and $e_{t-1}$ is the hidden state of the decoder at last time step. $h_{i}$ is the $i_{th}$ time step segment of $h$. $l_{w}$, $l_{c}$, and $l_{o}$ are linear functions. $\sigma$ is a nonlinear activation function. $k$ is the GRU function. $o_{t}$ is the output and $e_{t}$ is the hidden state of the GRU. $p(d_{t}|u,s)$ is the probability distribution of the vocabulary at time step ${t}$. The attention weights ${w_{t}}$ are computed based on the decoder’s hidden state and the generated word at time step ${t-1}$. Then the computed weights are applied to the concatenated hidden states $h$ of encoders. The result $a_{t}$ is the context vector for the decoder at time step $t$. The context vector and the last generated word are combined by a linear function $l_{c}$ followed by a nonlinear activation function. The result $b_{t}$ is concatenated with the encoded tweet context $c_{u}$, and then fed into GRU together with the decoder’s last hidden state $e_{t-1}$. Finally, the probability of each vocabulary word is computed from $o_{t}$.

4.2 Variational Decipher

The Variational Decipher is based on the CVAE model, which is another model that can be used to parametrize the conditional probability $p(d^{*}|(u,s))$ in the objective function (Equation 1). Unlike the Seq2Seq model, which directly parametrizes $p(d^{*}|(u,s))$, our variational decipher formulates the task as follows:

[TABLE]

where $z$ is the latent variable. $p(d^{*}|(u,s)$ is written as the marginalization of the product of two terms over the latent space. Since the integration over $z$ is intractable, we instead try to maximize the evidence lower bound (ELBO). Our variational lower bound objective is in the following form:

[TABLE]

where $p_{\varphi}(d^{*}|z,u,s)$ is the likelihood, $p_{\alpha}(z|d^{*},u,s)$ is the posterior, $p_{\beta}(z|u,s)$ is the prior, and $D_{KL}$ is the Kullback-Leibler (KL) divergence. We use three neural networks to model these three probability distributions. An overview of our variational decipher is shown in Figure 3. We first use four recurrent neural networks to encode the (tweet, symbol, definition) pair in the dataset. Similar to what we do in the Seq2Seq model, there are two encoders for the hate symbol. One is at the word level and the other is at the character level. The encoding of symbols and tweets are exactly the same as in our Seq2Seq model (see Equations 2-4). The difference is that we also need to encode definitions for the Variational Decipher.

[TABLE]

Here, $f_{d}$ is the LSTM function. $x$ is the output of the LSTM at the last time step and $h_{d}$ is the hidden state of the LSTM at all time steps. The condition vector $c$ is the concatenation of the encoded symbol words, symbol characters, and the tweet text:

[TABLE]

We use multi-layer perceptron (MLP) to model the posterior and the prior in the objective function. The posterior network and the prior network have the same structure and both output a probability distribution of latent variable $z$. The only difference is that the input of the posterior network is the concatenation of the encoded definition $x$ and the condition vector $c$ while the input of the prior network is only the condition vector $c$. Therefore, the output of the posterior network $p=p_{\alpha}(z|d^{*},u,s)$ and the output of the prior network $p^{\prime}=p_{\beta}(z|u,s)$. By assuming the latent variable $z$ has a multivariate Gaussian distribution, the actual outputs of the posterior and prior networks are the mean and variance: ($\mu$, $\Sigma$) for the posterior network and ($\mu^{\prime}$, $\Sigma^{\prime}$) for the prior network.

[TABLE]

$g$ is the MLP function of the posterior network and $g^{\prime}$ is that of the prior network. During training, the latent variable $z$ is randomly sampled from the Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ and fed into the likelihood network. During testing, the posterior network is replaced by the prior network, so $z$ is sampled from $\mathcal{N}(\mu^{\prime},\Sigma^{\prime})$. The likelihood network is modeled by an RNN decoder with attention mechanism, very similar to the decoder of our Seq2Seq model. The only difference lies in the input for the GRU. The decoder in our Variational Decipher model is to model the likelihood $p_{\varphi}(d^{*}|z,u,s)$, which is conditioned on the latent variable, tweet context, and the symbol. Therefore, for the Variational Decipher, the condition vector $c$ and the sampled latent variable $z$ are fed into the decoder.

[TABLE]

$e_{t-1}$ is the hidden state of the RNN decoder at the last time step. $k$ is the GRU function. $o_{t}$ is its output and $e_{t}$ is its hidden state. Detailed decoding process and explanations are in section 4.1.

According to the objective function in Equation 12, the loss function of the Variational Decipher is as follows:

[TABLE]

It consists of two parts. The first part $L_{REC}$ is called reconstruction loss. Optimizing $L_{REC}$ can push the sentences generated by the posterior network and the likelihood network closer to the given definitions. The second part $L_{KL}$ is the KL divergence loss. Optimizing this loss can push the output Gaussian Distributions of the prior network closer to that of the posterior network. This means we teach the prior network to learn the same knowledge learned by the posterior network, such that during testing time, when the referential definition $d^{*}$ is no longer available for generating the latent variable $z$, the prior network can still output a reasonable probability distribution over the latent variable $z$. The complete training and testing process for the Variational Decipher is shown in Algorithm 1. $M$ is the predefined maximum length of the generated text. $BCE$ refers to the Binary Cross Entropy loss.

5 Experiments

5.1 Experimental Settings

We use the dataset collected as described in section 3 for training and testing. We randomly selected 2,440 tuples for testing and use the remaining 16,227 tuples for training. Note that there are no overlapping hate symbols between the training dataset $U$ and the testing dataset $D$.

We split the 2,440 tuples of the testing dataset $D$ into two separate parts, $D_{s}$ and $D_{d}$. $D_{s}$ consists of 1,681 examples and $D_{d}$ consists of 759 examples. In the first testing dataset $D_{s}$, although each hate symbol does not appear in the training dataset, the corresponding definition appears in the training dataset. In the second testing dataset $D_{d}$, neither the hate symbols nor the corresponding definitions appear in the training dataset. We do this split because deciphering hate symbols in these two cases has different levels of difficulty.

This split criterion means that for each hate symbol in $D_{s}$, there exists some symbol in the training dataset that has the same meaning but in different surface forms. For example, the hate symbol wigwog and Wig Wog have the same definition but one is in the training dataset, the other is in the first testing dataset. We assume that such types of hate symbols share similar surface forms or similar tweet contexts. Therefore, the first testing dataset $D_{s}$ is to evaluate how well the model captures the semantic similarities among the tweet contexts in different examples or the similarities among different surface forms of a hate symbol.

Deciphering the hate symbols in the second testing dataset $D_{d}$ is more challenging. Both the unseen hate symbols and definitions require the model to have the ability to accurately capture the semantic information in the tweet context and then make a reasonable prediction. The second testing dataset $D_{d}$ is used to evaluate how well the model generalizes to completely new hate symbols.

For the Seq2Seq model, we use negative log-likelihood loss for training. Both models are optimized using Adam optimizer  Kingma and Ba (2015). The hyper-parameters of two models are exactly the same. We set the maximum generation length $M=50$. The hidden size of the encoders is 64. The size of the word embedding is 200 and that of character embedding is 100. The word embeddings and character embeddings are randomly initialized. Each model is trained for 50 epochs. We report the deciphering results of two models on three testing datasets $D$, $D_{s}$ and $D_{d}$.

5.2 Experimental Results

Quantitative Results: We use equally weighted BLEU score for up to 4-grams  Papineni et al. (2002), ROUGE-L  Lin (2004) and METEOR  Banerjee and Lavie (2005) to evaluate the decipherment results. The results are shown in Table 1. Figure 4 shows the BLEU score achieved by the two models on three testing datasets $D$, $D_{s}$ and $D_{d}$ during the training process. Both our Seq2Seq model and Variational Decipher achieve reasonable BLEU scores on the testing datasets. The Seq2Seq model outperforms the Variational Decipher on $D_{s}$ while Variational Decipher outperforms Seq2Seq on $D_{d}$. Note that $D_{s}$ is more than twice the size of $D_{d}$. Therefore, Seq2Seq outperforms Variational Decipher on the entire testing dataset $D$. The different performance of the two models on $D_{s}$ and $D_{d}$ is more obvious in Figure 4. The gap between the performance of the Seq2Seq model on $D_{s}$ and $D_{d}$ is much larger than that between the performance of the Variational Decipher on these two datasets.

Human Evaluation: We employed crowd-sourced workers to evaluate the deciphering results of two models. We randomly sampled 100 items of deciphering results from $D_{s}$ and another 100 items from $D_{d}$. Each item composes a choice question and each choice question is assigned to five workers on Amazon Mechanical Turk. In each choice question, the workers are given the hate symbol, the referential definition, the original tweet and two machine-generated plain texts from the Seq2Seq model and Variational Decipher. Workers are asked to select the more reasonable of the two results. In each choice question, the order of the results from the two models is permuted. Ties are permitted for answers. We batch five items in one assignment and insert an artificial item with two identical outputs as a sanity check. The workers who fail to choose “tie” for that item are rejected from our test. The human evaluation results are shown in Table 2, which coincide with the results in Table 1 and Figure 4.

Discussion: When deciphering the hate symbols that have the same definitions as in the training dataset, the model can rely more on the surface forms of hate symbols than the tweet context to make a prediction because usually the hate symbols that share the same definitions also have similar surface forms. However, when it comes to the hate symbols with unseen definitions, simply relying on the surface forms cannot lead to a reasonable deciphering result. Instead, the model should learn the relationships between the context information and the definition of the symbol. Therefore, the different performances of two models on the two testing datasets $D_{s}$ and $D_{d}$ indicate that the Seq2Seq model is better at capturing the similarities among different surface forms of a hate symbol, while the Variational Decipher is better at capturing the semantic relationship between the tweet context and the hate symbol. The Sequence-to-Sequence model tries to capture such kinds of relationships by compressing all the context information into a fixed length vector, so its deciphering strategy is actually behavior cloning. On the other hand, the Variational Decipher captures such relationships by explicitly modeling the posterior and likelihood distributions. The modeled distributions provide higher-level semantic information compared to the compressed context, which allows the Variational Decipher to generalize better to the symbols with unseen definitions. This explains why the gap between the performance of the Seq2Seq model on two datasets is larger.

5.3 Error Analysis

Figure 5 shows some example errors of the deciphering results of our Seq2Seq model and Variational Decipher. One problem with the deciphering results is that the generated sentences have poor grammatical structure, as shown in Figure 5. This is mainly because the size of our dataset is small, and the models need a much larger corpus to learn the grammar. We anticipate that the generation performance will be improved with a larger dataset.

For the hate symbols in $D_{s}$, the deciphering results are of high quality when the length of referential definitions are relatively short. An example is macaca, a French slur shows in Figure 5. The deciphering result of the Seq2Seq model is close to the referential definition. As to the Variational Decipher, although the result is not literally the same as the definition, the meaning is close. closet homosexuals in Figure 5 is another example. However, when the length of the referential definition increases, the performance of both models tends to be unsatisfactory, as the third example confederate flag shows in Figure 5. Although there exists the symbol Confederate Flag with the same definition in the training set, both models fail on this example. One possible reason is that the complexity of generating the referential definition grows substantially with the increasing length, so when the tweet context and the symbol itself cannot provide enough information, the generation model cannot learn the relationship between the symbol and its definition.

Deciphering hate symbols in $D_{d}$ is much more challenging. Even for humans, deciphering completely new hate symbols is not a simple task. The two examples in Figure 5 show that the models have some ability to capture the semantic similarities. For the symbol niggering, the Variational Decipher generates the word nigger and Seq2Seq model generates black. For Heil Hitler, the Variational Decipher generates leader person and Nazi, while Seq2Seq also generates Nazi. Although these generated words are not in the definition, they still make some sense.

6 Conclusion

We propose a new task of learning to decipher hate symbols and create a symbol-rich tweet dataset. We split the testing dataset into two parts to analyze the characteristics of the Seq2Seq model and the Variational Decipher. The different performance of these two models indicates that the models can be applied to different scenarios of hate symbol deciphering. The Seq2Seq model outperforms the Variational Decipher for deciphering the hate symbols with similar definitions to that in the training dataset. This means the Seq2Seq model can better explain the hate symbols when Twitter users intentionally misspell or abbreviate common slur terms. On the other hand, the Variational Decipher tends to be better at deciphering hate symbols with unseen definitions, so it can be applied to explain newly created hate symbols on Twitter. Although both models show promising deciphering results, there still exists much room for improvement.