GANs for Semi-Supervised Opinion Spam Detection

TL;DR

This paper introduces spamGAN, a generative adversarial network that effectively detects opinion spam using limited labeled data and unlabeled reviews, outperforming existing methods and generating realistic reviews.

Contribution

The paper presents spamGAN, a novel GAN-based approach that enhances opinion spam detection with minimal labeled data and also generates plausible reviews.

Findings

spamGAN outperforms existing spam detection methods with limited labeled data

spamGAN can generate reviews with reasonable perplexity

The approach improves text classification in opinion spam detection

Abstract

Online reviews have become a vital source of information in purchasing a service (product). Opinion spammers manipulate reviews, affecting the overall perception of the service. A key challenge in detecting opinion spam is obtaining ground truth. Though there exists a large set of reviews online, only a few of them have been labeled spam or non-spam. In this paper, we propose spamGAN, a generative adversarial network which relies on limited set of labeled data as well as unlabeled data for opinion spam detection. spamGAN improves the state-of-the-art GAN based techniques for text classification. Experiments on TripAdvisor dataset show that spamGAN outperforms existing spam detection techniques when limited labeled data is used. Apart from detecting spam reviews, spamGAN can also generate reviews with reasonable perplexity.

Tables

Table 1. Table 1: Accuracy (Mean ± plus-or-minus \pm Std) for Different % Labeled Data

Method	10% Labeled	30%	50%	70%	90%	100%
spamGAN-0%	0.700 $\pm$ 0.02	0.811 $\pm$ 0.02	0.838 $\pm$ 0.01	0.845 $\pm$ 0.01	0.852 $\pm$ 0.02	0.862 $\pm$ 0.01
spamGAN-50%	0.678 $\pm$ 0.03	0.797 $\pm$ 0.03	0.839 $\pm$ 0.02	0.845 $\pm$ 0.02	0.857 $\pm$ 0.02	0.856 $\pm$ 0.01
spamGAN-70%	0.695 $\pm$ 0.05	0.780 $\pm$ 0.03	0.828 $\pm$ 0.02	0.850 $\pm$ 0.01	0.841 $\pm$ 0.02	0.844 $\pm$ 0.02
spamGAN-100%	0.681 $\pm$ 0.02	0.783 $\pm$ 0.02	0.831 $\pm$ 0.01	0.837 $\pm$ 0.01	0.843 $\pm$ 0.02	0.845 $\pm$ 0.01
Base classifier	0.722 $\pm$ 0.03	0.786 $\pm$ 0.02	0.791 $\pm$ 0.02	0.829 $\pm$ 0.01	0.824 $\pm$ 0.02	0.827 $\pm$ 0.02
DRI-RCNN	0.647 $\pm$ 0.10	0.757 $\pm$ 0.01	0.796 $\pm$ 0.01	0.834 $\pm$ 0.18	0.835 $\pm$ 0.02	0.846 $\pm$ 0.01
RCNN	0.538 $\pm$ 0.09	0.665 $\pm$ 0.14	0.733 $\pm$ 0.09	0.811 $\pm$ 0.03	0.834 $\pm$ 0.02	0.825 $\pm$ 0.02
Co-Train (Naive Bayes)	0.655 $\pm$ 0.01	0.740 $\pm$ 0.01	0.738 $\pm$ 0.02	0.743 $\pm$ 0.01	0.754 $\pm$ 0.01	0.774 $\pm$ 0.01
PU Learn (Naive Bayes)	0.508 $\pm$ 0.02	0.713 $\pm$ 0.03	0.816 $\pm$ 0.01	0.826 $\pm$ 0.01	0.838 $\pm$ 0.02	0.843 $\pm$ 0.02

Table 2. Table 2: F1-Score (Mean ± plus-or-minus \pm Std) for Different % Labeled Data

Method	10% Labeled	30%	50%	70%	90%	100%
spamGAN-0%	0.718 $\pm$ 0.02	0.812 $\pm$ 0.02	0.840 $\pm$ 0.01	0.848 $\pm$ 0.02	0.854 $\pm$ 0.02	0.868 $\pm$ 0.01
spamGAN-50%	0.674 $\pm$ 0.05	0.797 $\pm$ 0.03	0.843 $\pm$ 0.01	0.848 $\pm$ 0.02	0.860 $\pm$ 0.02	0.863 $\pm$ 0.01
spamGAN-70%	0.702 $\pm$ 0.05	0.784 $\pm$ 0.03	0.830 $\pm$ 0.02	0.856 $\pm$ 0.01	0.848 $\pm$ 0.02	0.854 $\pm$ 0.01
spamGAN-100%	0.684 $\pm$ 0.03	0.788 $\pm$ 0.03	0.839 $\pm$ 0.02	0.844 $\pm$ 0.01	0.846 $\pm$ 0.02	0.850 $\pm$ 0.01
Base classifier	0.731 $\pm$ 0.03	0.795 $\pm$ 0.03	0.803 $\pm$ 0.02	0.829 $\pm$ 0.01	0.832 $\pm$ 0.02	0.838 $\pm$ 0.02
DRI-RCNN	0.632 $\pm$ 0.07	0.754 $\pm$ 0.02	0.779 $\pm$ 0.00	0.812 $\pm$ 0.03	0.817 $\pm$ 0.03	0.833 $\pm$ 0.02
RCNN	0.638 $\pm$ 0.01	0.715 $\pm$ 0.01	0.754 $\pm$ 0.02	0.776 $\pm$ 0.05	0.820 $\pm$ 0.03	0.833 $\pm$ 0.02
Co-Train (Naive Bayes)	0.637 $\pm$ 0.02	0.698 $\pm$ 0.01	0.680 $\pm$ 0.02	0.677 $\pm$ 0.01	0.712 $\pm$ 0.01	0.726 $\pm$ 0.01
PU-Learn (Naive Bayes)	0.050 $\pm$ 0.02	0.636 $\pm$ 0.05	0.815 $\pm$ 0.02	0.837 $\pm$ 0.02	0.844 $\pm$ 0.02	0.852 $\pm$ 0.01

Equations

$$\text{vspace}-2mm\mathcal{G}(y_{1\text{mathchar}58T}|z,c,{\theta }_g)=\prod _{t=1}^T\mathcal{G}(y_t|y_{1\text{mathchar}58t-1},z,\mathfrak{c},{\theta }_g)$$

$$\text{vspace}-2.0mm{\mathcal{L}}_{MLE}^{\mathcal{G}}=-\sum _{t=1}^T\log \mathcal{G}(y_t|y_{1\text{mathchar}58t-1},z,\mathfrak{c},{\theta }_g)$$

$$\text{vspace}-2.0mm\begin{array}{rl}\mathcal{D}(y_{1\text{mathchar}58T}|{\theta }_d)& =\frac{1}{T}\sum _{t=1}^T{Q}_{\mathcal{D}}(y_{1\text{mathchar}58t-1},y_t)\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}^{(\mathcal{D})}=& {\mathbb{E}}_{y_{1\text{mathchar}58T}\sim P_R}-\left[\log \mathcal{D}(y_{1\text{mathchar}58T}|{\theta }_d)\right]+{\mathbb{E}}_{y_{1\text{mathchar}58T}\sim \mathcal{G}}-\left[\log (1-\mathcal{D}(y_{1\text{mathchar}58T}|{\theta }_d))\right]\end{array}$$

$$V_{\mathcal{D}}(y_{1\text{mathchar}58t-1})={\mathbb{E}}_{y_t}\left[Q_{\mathcal{D}}(y_{1\text{mathchar}58t-1},y_t)\right]$$

$$\begin{array}{rl}{\mathcal{L}}^{({\mathcal{D}}_{\text{crit}})}& ={\mathbb{E}}_{y_{1\text{mathchar}58T}}\sum _{t=1}^T{\Vert Q_{\mathcal{D}}(y_{1\text{mathchar}58t-1},y_t)-V_{\mathcal{D}}(y_{1\text{mathchar}58t-1})\Vert }^2\end{array}$$

$$\begin{array}{rl}\mathcal{C}(y_{1\text{mathchar}58T},c|{\theta }_c)& =\frac{1}{T}\sum _{t=1}^T{Q}_{\mathcal{C}}(y_{1\text{mathchar}58t-1},y_t,c)\end{array}$$

$$\text{vspace}-2.0mm\begin{array}{rl}{\mathcal{L}}^{\mathcal{C}}& ={\mathcal{L}}^{({\mathcal{C}}_{\text{R}})}+{\mathcal{L}}^{({\mathcal{C}}_{\text{G}})}\end{array}\text{vspace}-2.0mm$$

$$\begin{array}{rl}{\mathcal{L}}^{({\mathcal{C}}_{\text{R}})}& ={\mathbb{E}}_{(y_{1\text{mathchar}58T},c)\sim P_R(y,\mathfrak{c})}\left[-\log \mathcal{C}(c|y_{1\text{mathchar}58T},{\theta }_c)\right]\\ {\mathcal{L}}^{({\mathcal{C}}_{\text{G}})}& ={\mathbb{E}}_{c\sim P_c,y_{1\text{mathchar}58T}\sim \mathcal{G}}\left[-\log \mathcal{C}(c|y_{1\text{mathchar}58T},{\theta }_c)-\beta \mathcal{H}(\mathcal{C}(c|y_{1\text{mathchar}58T},{\theta }_C))\right]\end{array}$$

$$V_{\mathcal{C}}(y_{1\text{mathchar}58t-1,}c)={\mathbb{E}}_{y_t}[Q_{\mathcal{C}}(y_{1\text{mathchar}58t-1},y_t,c)]$$

$$\begin{array}{rl}{\mathcal{L}}^{({\mathcal{C}}_{\text{crit}})}& ={\mathbb{E}}_{y_{1\text{mathchar}58T}}\sum _{t=1}^T{\Vert Q_{\mathcal{C}}(y_{1\text{mathchar}58t-1},y_t,c)-V_{\mathcal{C}}(y_{1\text{mathchar}58t-1,}c)\Vert }^2\end{array}$$

$$R(y_{1\text{mathchar}58T})=2\cdot \frac{\mathcal{D}(y_{1\text{mathchar}58T}|{\theta }_d)\cdot \mathcal{C}(y_{1\text{mathchar}58T},c|{\theta }_c)}{\mathcal{D}(y_{1\text{mathchar}58T}|{\theta }_d)+\mathcal{C}(y_{1\text{mathchar}58T},c|{\theta }_c)}$$

$$\begin{array}{rl}{\mathcal{L}}^{(\mathcal{G})}& ={\mathbb{E}}_{y_{1\text{mathchar}58T}\sim \mathcal{G}}\left[R(y_{1\text{mathchar}58T})\right]\end{array}$$

$$\begin{array}{rl}& Q(y_{1\text{mathchar}58t},c)=2\cdot \frac{Q_{\mathcal{D}}(y_{1\text{mathchar}58t-1},y_t)\cdot Q_{\mathcal{C}}(y_{1\text{mathchar}58t-1},y_t,c)}{Q_{\mathcal{D}}(y_{1\text{mathchar}58t-1},y_t)+Q_{\mathcal{C}}(y_{1\text{mathchar}58t-1},y_t,c)}\\ & V(y_{1\text{mathchar}58t-1},c)=2\cdot \frac{V_{\mathcal{D}}(y_{1\text{mathchar}58t-1})\cdot V_{\mathcal{C}}(y_{1\text{mathchar}58t-1,}c)}{V_{\mathcal{D}}(y_{1\text{mathchar}58t-1})+V_{\mathcal{C}}(y_{1\text{mathchar}58t-1,}c)}\end{array}$$

$${\nabla }_{{\theta }_g}{\mathcal{L}}^{(\mathcal{G})}={\mathbb{E}}_{y_{1\text{mathchar}58T}}\sum _t^T$$

$$\times {\nabla }_{{\theta }_g}\log \mathcal{G}(y_t|y_{1\text{mathchar}58t-1},z,c,{\theta }_g)\text{vspace}-3.5mm$$

Full text

1 Introduction

Opinion spam is a widespread problem in e-commerce, social media, travel sites, movie review sites, etc. Jindal et al. (2010). Statistics show that more than $90\%$ of the consumers read reviews before making a purchase Hub (2018). The likelihood of purchase is also reported to increase when there are more reviews. Opinion spammers try to exploit such financial gains by providing spam reviews which influence readers and thereby affect sales. We consider the problem of identifying spam reviews as a classification problem, i.e., given a review, it needs to be classified either as spam or non-spam.

One of the main challenges in identifying spam reviews is the lack of labeled data, i.e., spam and non-spam labels Rayana and Akoglu (2015). While there exists a corpus of online reviews only few of them are labeled. This is mainly because manual labeling is often time consuming, costly and subjective Li et al. (2018). Research shows that unlabeled data, when used in conjunction with small amounts of labeled data can produce considerable improvement in learning accuracy Ott et al. (2011). There is very limited research on using semi-supervised learning techniques for opinion spam detection Crawford et al. (2015). The existing semi-supervised learning approaches Li et al. (2011); Hernández et al. (2013); Li et al. (2014) for identifying opinion spam use pre-defined set of features for training their classifier. In this paper, we will use deep neural networks which automatically discovers features needed for classification LeCun et al. (2015).

Deep generative models have shown promising results for semi-supervised learning Kumar et al. (2017). Specifically, Generative Adversarial Networks (GANs) Goodfellow et al. (2014) which have the ability to generate samples very close to real data, have achieved state-of-the art results. However, most research on GANs are for images (continuous values) and not text data (discrete values) Fedus et al. (2018).

GANs operate by training two neural networks which play a min-max game: discriminator D tries to discriminate real training samples from fake ones and generator G tries to generate fake training samples to fool the discriminator. The main drawback with GANs is that: 1) when the data is discrete, the gradient from the discriminator may not be useful for improving the generator. This is because, the slight change in weights brought forth by the gradients may not correspond to a suitable discrete mapping in the dictionary Huszár (2015); 2) the discrimination is based on the entire sentence not parts of it, giving rise to the sparse rewards problem Yu et al. (2017).

Existing works on GANs for text data generation are limited by the length of the sentence that can be generated, e.g., MaskGAN Fedus et al. (2018) considers $40$ words per sentence. These approaches may not be suitable for processing most online reviews, which are relatively lengthy. For example, the TripAdvisor review dataset used in our experiments has sentences with median length $132$. Further, GANs have also not been fully investigated for text classification tasks.

In this paper, we propose spamGAN, a semi-supervised GAN based approach for classifying opinion spam. spamGAN uses both labeled instances and unlabeled data to correctly learn the input distribution, resulting in better prediction accuracy for comparatively longer reviews. spamGAN consists of $3$ different components: generator, discriminator, classifier which work together to not only classify spam reviews but also generate samples close to the train set. We conduct experiments on TripAdvisor dataset and show that spamGAN outperforms existing works when using limited labeled data.

Following are the main contributions of this paper: 1) we propose spamGAN: a semi-supervised GAN based model to detect opinion spam. To the best of our knowledge, we are the first to explore the potential of GANs for spam detection; 2) the proposed GAN model improves the state-of-the-art GAN based models for semi-supervised text classification; 3) most existing research on opinion spam (other than deep learning methods) manually identify heuristics/features for classifying spamming behavior, however in our GAN based approach, the features are learned by the neural network; 4) experiments show that spamGAN outperforms state-of-the art methods in classifying spam when limited labeled data is used; 5) spamGAN can also generate spam/non-spam reviews very similar to the training set which can be used for synthetic data generation in cases with limited ground truth.

2 Related Work

Most existing opinion spam detection techniques are supervised methods based on pre-defined features. Jindal and Liu (2008) used logistic regression with product, review and reviewer-centric features. Ott et al. (2011) used n-gram features to train a Naive Bayes and SVM classifier. Feng et al. (2012); Mukherjee et al. (2013); Li et al. (2015) used part-of-speech tags and context free grammar parse trees, behavioral features, spatio-temproal features, respectively. Wang et al. (2011); Akoglu et al. (2013) used graph based algorithms.

Neural network methods for spam detection consider the reviews as input wihtout specific feature extraction. GRNN Ren and Ji (2017) used a gated recurrent neural network to study the contexual information of review sentences. DRI-RCNN Zhang et al. (2018) used a recurrent network for learning the contextual information of the words in the reviews. DRI-RCNN extends RCNN Lai et al. (2015) by learning embedding vectors with respect to both spam and non-spam labels for the words in the reviews. Since RCNN and DRI-RCNN use neural networks for spam classification, we will use these supervised methods for comparison in our experiments.

Few semi-supervised methods for opinion spam detection exist. Li et al. (2011) used co-training with Naive-Bayes classifier on reviewer, product and review features. Hernández et al. (2013); Li et al. (2014) used only positively labeled samples along with unlabeled data. Rayana and Akoglu (2015) used review features, timestamp, ratings as well as pairwise markov random field network of reviewers and product to build a supervised algorithm along with semi-supervised extensions. Other un-supervised methods for spam detection Xu et al. (2015) exists, but, they are out of the scope of this work.

The ongoing research on GANs for text classification aim to address the drawbacks of GANs in generating sentences with respect to the gradients and the sparse rewards problem. SeqGAN Yu et al. (2017) addresses them by considering sequence generation as a reinforcement learning problem. Monte Carlo Tree Search (MCTS) is used to overcome the issue of sparse rewards, however it is computationally intractable. StepGAN Tuan and Lee (2018) and MaskGAN Fedus et al. (2018) use the actor-critic Konda and Tsitsiklis (2000) method to learn the rewards, however MaskGAN is limited by length of the sequence. Further, all of them focus on sentence generation. CSGAN Li et al. (2018) deals with sentence classification, but it uses MCTS and character-level embeddings. spamGAN differs from CSGAN in using the actor-critic reinforcement learning method for sequence generation and word-level embeddings, suitable for longer sentences.

3 spamGAN

In this section, we will present the problem set-up, the three components of spamGAN as well as their interactions through a sequential decision making framework.

3.1 Problem Set-up

Let $\mathbb{D_{L}}$ be the set of reviews labeled spam or non-spam. Given the cost of labeling, we hope to improve classification performance by also using $\mathbb{D_{U}}$, a significantly larger set of unlabeled reviews222$\mathbb{D_{U}}$ includes both spam/non-spam reviews.. Let $\mathbb{D}=\mathbb{D_{L}}\cup\mathbb{D_{U}}$ be a combination of labeled and unlabeled sentences for training333Training (see Alg. 1) can use only $\mathbb{D_{L}}$ or both $\mathbb{D_{L}}$ and $\mathbb{D_{U}}$.. Each training sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}T}=\{y_{1},y_{2},\ldots y_{t},\ldots,y_{T}\}$ consists of a sequence of $T$ word tokens, where $y_{t}\in\mathtt{Y}$ represents the $t^{th}$ token in the sentence and $\mathtt{Y}$ is a corpus of tokens used. For sentences belonging to $\mathbb{D_{L}}$, we also include a class label belonging to one of the $2$ classes $\mathfrak{c}\in\mathbb{C}\mathrel{\mathop{\mathchar 58\relax}}\{\mathtt{spam},\mathtt{non\text{-}spam}\}$.

To leverage both the labeled and unlabeled data, we include three components in spamGAN: the generator $\mathcal{G}$, the discriminator $\mathcal{D}$, and the classifier $\mathcal{C}$ as shown in Fig. 1. The generator, for a given class label, learns to generate new sentences (we call them $\mathtt{fake}$444Fake sentences are those produced by the generator. Spam sentences are deceptive sentences with class label $\mathtt{spam}$. Generator can generate fake sentences belonging to $\{\mathtt{spam}$ or $\mathtt{non\text{-}spam}\}$ class. sentences) similar to the real sentences in the train set belonging to the same class. The discriminator learns to differentiate between real and fake sentences, and informs the generator (via rewards) if the generated sentences are unrealistic. This competition between the generator and discriminator improves the quality of the generated sentence.

We know the class labels for the fake sentences produced by the generator as they are controlled Hu et al. (2017), i.e., constrained by class labels $\{\mathtt{spam},\mathtt{non\text{-}spam}\}$. The classifier is trained using real labeled sentences from $\mathbb{D_{L}}$ and fake sentences produced by the generator, thus improving its ability to generalize beyond the small set of labeled sentences. The classifier’s performance on fake sentences is also used as feedback to improve the generator: better classification accuracy results in more rewards. While the discriminator and generator are competing, the classifier and generator are mutually bootstrapping. As the $3$ components of spamGAN are trained, the generator produces sentences very similar to the training set while the classifier learns the characteristics of spam and non-spam sentences in order to identify them correctly.

3.2 Generator

If $P_{R}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T},\mathfrak{c})$ is the true joint distribution of sentences $y_{1\mathrel{\mathop{\mathchar 58\relax}}T}$ and classes $\mathfrak{c}\in\mathbb{C}$ from the real training set, the generator aims to find a parameterized conditional distribution $\mathcal{G}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|z,c,\theta_{g})$ that best approximates the true distribution. The generated fake sentence is conditioned on the network parameters $\theta_{g}$, noise vector $z$, and class label $c$, which are sampled from the prior distribution $P_{z}$ and $P_{\mathfrak{c}}$, respectively. $z$ and $c$ together make up the context vector. The context vector is concatenated to the generated sentence at every timestep Tuan and Lee (2018), ensuring that the actual class labels for each generated fake sentence is retained.

While sampling from $\mathcal{G}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|z,c,\theta_{g})$, the word tokens are generated auto-regressively, decomposing the distribution over token sequences into the ordered conditional sequence,

[TABLE]

During pre-training, we use batches of real sentences from $\mathbb{D}$ and minimize the cross-entropy of the next token conditioned on the preceding ones. Specifically, we minimize the loss (Eqn. 2) over real sentence-class pairs $(y_{1\mathrel{\mathop{\mathchar 58\relax}}T},\mathfrak{c})$ from $\mathbb{D_{L}}$ as well as unlabeled real sentences from $\mathbb{D_{U}}$ with randomly-assigned class labels drawn from the class prior distribution.

[TABLE]

During adversarial training, we treat sequence generation as a sequential decision making problem Yu et al. (2017). The generator acts as a reinforcement learning agent and is trained to maximize the expected rewards using policy gradients, where the rewards are feedback obtained from the discriminator and classifier for the generated sentences (See Sec. 3.5). For implementation, we use a unidirectional multi-layer recurrent neural network with gated recurrent units as the base cell to represent the generator.

3.3 Discriminator

The discriminator $\mathcal{D}$, with parameters $\theta_{d}$ predicts if a sentence is real (sampled from $P_{R}$) or fake (produced by the generator) by computing a probability score $\mathcal{D}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|\theta_{d})$ that the sentence is real. Like Tuan and Lee (2018) instead of computing the score at the end of the sentence, the discriminator produces scores for every timestep $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$, which are then averaged to produce the overall score.

[TABLE]

$Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$ is the intermediate score for timestep $t$ and is based solely on the preceding partial sentence, $y_{1\mathrel{\mathop{\mathchar 58\relax}}t}$. In a setup reminiscent of $Q$-learning, we consider $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$ to be the estimated value for the state $s=y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1}$ and action $a=y$. Thus, the discriminator provides estimates for the true state-action values without the additional computational overhead of using MCTS rollouts.

We train the discriminator like traditional GANs by maximizing the score $\mathcal{D}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|\theta_{d})$ for real sentences and minimizing it for fake ones. This is achieved by minimizing the loss $\mathcal{L^{(D)}}$,

[TABLE]

We also include a discrimination critic $\mathcal{D}_{crit}$ Konda and Tsitsiklis (2000) which is trained to approximate the score $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$ from the discriminator network, for the next token $y_{t}$ based on the preceding partial sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1}$. The approximated score $V_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1})$ will be used to stabilize policy gradient updates for the generator during adversarial training.

[TABLE]

$\mathcal{D}_{crit}$ is trained to minimize the sequence mean-squared error between $V_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1})$ and the actual score $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$.

[TABLE]

The discriminator network is implemented as a unidirectional Recurrent Neural Network (RNN) with one dense output layer which produces the probability that a sentence is real at each timestep, i.e., $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$. For the discrimination critic, we have a additional output dense layer (different from the one that computes $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$) attached to the discriminator RNN, which estimates $V_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1})$ for each timestep.

3.4 Classifier

Given a sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}T}$, the classifier $\mathcal{C}$ with parameters $\theta_{c}$ predicts if the sentence belongs to class $c\in\mathbb{C}$. Like the discriminator, it assigns a prediction score at each timestep $Q_{\mathcal{C}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t},c)$ for the partial sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}t}$, which identifies the probability the sentence belongs to class $c$. The intermediate scores are then averaged to produce the overall score:

[TABLE]

The classifier loss $\mathcal{L^{C}}$ is based on: 1) $\mathcal{L^{(C_{\text{R}})}}$, the cross-entropy loss on true labeled sentences computed using the overall classifier sentence score; 2) $\mathcal{L^{(C_{\text{G}})}}$ the loss for the fake sentences. Fake sentences are considered as potentially-noisy training examples, so we not only minimize cross-entropy loss but also include Shannon entropy $\mathcal{H}(\mathcal{C}(c|y_{1\mathrel{\mathop{\mathchar 58\relax}}T},\theta_{C}))$.

[TABLE]

[TABLE]

In $\mathcal{L^{(C_{\text{G}})}}$, $\beta$, the balancing parameter, influences the impact of Shannon entropy. Including $\mathcal{H}(\mathcal{C}(c|y_{1\mathrel{\mathop{\mathchar 58\relax}}T},\theta_{C}))$, for minimum entropy regularization Hu et al. (2017), allows the classifier to predict classes for generated fake sentences more confidently. This is crucial in reinforcing the generator to produce sentences of the given class during adversarial training.

Like in discriminator, we include a classification critic $\mathcal{C}_{crit}$ to estimate the classifier score $Q_{\mathcal{C}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t},c)$ for $y_{t}$ based on the preceding partial sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1}$,

[TABLE]

The implementation of the classifier is similar to the discriminator. We use a unidirectional recurrent neural network with a dense output layer producing the predicted probability distribution over classes $\mathfrak{c}\in\mathbb{C}$. The classification critic is also an alternative head off the classifier RNN with an additional dense layer estimating $V_{\mathcal{C}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1,}c)$ for each timestep. We train this classifier critic by minimizing $\mathcal{L^{(C\text{crit})}}$,

[TABLE]

3.5 Reinforcement Learning Component

We consider a sequential decision making framework, in which the generator acts as as a reinforcement learning agent. The current state of the agent is the generated tokens $s_{t}=y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1}$ so far. The action $y_{t}$ is the next token to be generated, which is selected based on the stochastic policy $\mathcal{G}(y_{t}|y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},z,c,\theta_{g})$. The reward the agent receives for the generated sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}T}$ of a given class $\mathfrak{c}$ is determined by the discriminator and classifier. Specifically, we take the overall scores $\mathcal{D}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|\theta_{d})$ (Eqn.3) and $\mathcal{C}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T},c|\theta_{c})$ (Eqn. 7) and blend them in a manner reminiscent of the F1 score, producing the sentence reward,

[TABLE]

This reward $R(y_{1\mathrel{\mathop{\mathchar 58\relax}}T})$ is for the entire sentence delivered during the final timestep, with reward for every other timestep being zero Tuan and Lee (2018). Thus, the generator agent seeks to maximize the expected reward, given by,

[TABLE]

To maximize $\mathcal{L^{(G)}}$, the generator parameters $\theta_{g}$ are updated via policy gradients Sutton et al. (2000). Specifically, we use the advantage actor-critic method to solve for optimal policy Konda and Tsitsiklis (2000). The expectation in Eqn. 12 can be re-written using rewards for intermediate time-steps from the discriminator and classifier. The intermediate scores from the discriminator, $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t})$ and the classifier, $Q_{\mathcal{C}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t},c)$, are combined as shown in Eqn. 13 and the combined values serve as estimators for $Q(y_{1\mathrel{\mathop{\mathchar 58\relax}}t},c)$, the expected reward for sentence $y_{1\mathrel{\mathop{\mathchar 58\relax}}t}$. To reduce variance in the gradient estimates, we replace $Q(y_{1\mathrel{\mathop{\mathchar 58\relax}}t},c)$ by the advantage function $Q(y_{1\mathrel{\mathop{\mathchar 58\relax}}t},c)-V(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},c)$, where $V(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},c)$ is given by Eqn. 13. We use $\alpha=T-t$ in Eqn. 14 to increase the importance of initially-generated tokens while updating $\theta_{g}$. $\alpha$ is a linearly-decreasing factor which corrects the relative lack of confidence in the initial intermediate scores from the discriminator and classifier.

[TABLE]

During adversarial training, we perform gradient ascent to update the generator using the gradient equation shown below,

[TABLE]

3.6 Pre-Training

Before beginning adversarial training, we pre-train the different components of spamGAN. The generator $\mathcal{G}$ is pre-trained using maximum likelihood estimation (MLE) Grover et al. (2018) by updating the parameters via Eqn 2. Once the generator is pre-trained, we take batches of real sentences from the labeled dataset $\mathbb{D_{L}}$, the unlabeled dataset $\mathbb{D_{U}}$ and fake sentences sampled from $\mathcal{G}(y_{1\mathrel{\mathop{\mathchar 58\relax}}T}|z,c,\theta_{g})$ to pre-train the discriminator minimizing the loss $\mathcal{L^{(D)}}$ in Eqn. 4. The classifier $\mathcal{C}$ is pre-trained solely on real sentences from the labeled dataset $\mathbb{D_{L}}$. It is trained to minimize the cross-entropy loss $\mathcal{L^{(C_{\text{R}})}}$ on real sentences and their labels. The critic networks $\mathcal{D_{\text{crit}}}$ and $\mathcal{C_{\text{crit}}}$ are trained by minimizing their loses $\mathcal{L^{(D\text{crit})}}$ (Eqn. 6) and $\mathcal{L^{(C\text{crit})}}$ (Eqn. 10). Such pre-training addresses the problem of mode collapse Guo et al. (2018) to a satisfactory extent.

3.7 spamGAN algorithm

Alg. 1 describes spamGAN in detail. After pre-training, we perform adversarial training for $\mathtt{Training\text{-}epochs}$ (Lines $4$-$25$). We create a batch of fake sentences using generator $\mathcal{G}$ by sampling classes $c$ from prior $P_{c}$ (Lines $6$-$7$). We compute $Q(y_{1\mathrel{\mathop{\mathchar 58\relax}}t},c)$, $V(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},c)$ using Eqn. 13 for every timestep (Line $9$). The generator is then updated using policy gradient in Eqn. 14 (Line $10$). This process is repeated for $\mathtt{G\text{-}Adv\text{-}epochs}$. Like Li et al. (2017) the training robustness is greatly improved when the generator is updated using MLE via Eqn 2 on sentences from $\mathbb{D}$ (Lines $11$-$13$). We then train the discriminator using real sentences from $\mathbb{D_{L}}$, $\mathbb{D_{U}}$ as well as fake sentences from the generator (Lines $15$-$16$). The discriminator is updated using Eqn. 4 (Line $17$). We also train the discrimination critic, by computing $Q_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},y_{t}),V_{\mathcal{D}}(y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1})$ for the fake sentences and updating the gradients using Eqn. 6 (Line $18$-$19$). This process is repeated for $\mathtt{D\text{-}epochs}$. We perform a similar set of operations for the classifier (Lines $20$-$25$).

4 Experiments

We use the TripAdvisor labeled dataset Ott et al. (2011) 555http://myleott.com/op-spam.html, consisting of 800 truthful reviews on Chicago hotels as well as $800$ deceptive reviews obtained from Amazon Mechanical Turk. We remove a small number of duplicate truthful reviews, to get a balanced labeled dataset of 1596 reviews. We augment the labeled set with $32,297$ unlabeled TripAdvisor reviews for Chicago hotels 666http://times.cs.uiuc.edu/ wang296/Data/index.html. All reviews are converted to lower-case and tokenized at word level, with a vocabulary $\mathtt{Y}$ of $10000$. The maximum sequence length is $T=128$ words, close to the median review length of the full dataset.

$\mathtt{Y}$ also includes tokens $\mathtt{\braket{start}}$, $\mathtt{\braket{end}}$, $\mathtt{\braket{unk}}$, and $\mathtt{\braket{pad}}$. $\mathtt{\braket{start}}$, $\mathtt{\braket{end}}$ are added to the beginning, end of each sentence. Sentences smaller than $T$ are padded with $\mathtt{\braket{pad}}$ while longer ones are truncated, ensuring a consistent sentence length. $\mathtt{\braket{unk}}$ replaces out-of-vocabulary words.

In spamGAN, the generator consists of 2 GRU layers of 1024 units each and an output dense layer providing logits for the $10,000$ tokens. The generator, discriminator and classifier are trained using ADAM optimizer. All use variational dropout=$0.5$ between recurrent layers and word embeddings with dimension $50$. For generator, learning rate = $0.001$, weight decay =$1\times 10^{-7}$. Gradient clipping is set to a maximum global norm of $5$. The discriminator contains 2 GRU layers of 512 units each and a dense layer with a single scalar output and sigmoid activation. The discrimination critic is implemented as an alternative dense layer. Learning rate =$0.0001$ and weight decay =$1\times 10^{-4}$. The classifier is similar to discriminator. We set balancing coefficient $\beta=1$. The train time of spamGAN using a Tesla P4 GPU was $\sim 1.5$ hrs.

We use a $80-20$ train-test split on labeled data. We compare spamGAN with $2$ supervised methods which use recurrent networks: 1) DRI-RCNN Zhang et al. (2018); 2) RCNN Lai et al. (2015) as well as $2$ semi-supervised methods: 3) Co-Training Li et al. (2011) with Naive Bayes classifier; 4) PU Learning Hernández et al. (2013) with Naive Bayes (SVM performed poorly) using only spam and unlabeled reviews.

We conduct experiments with $10,30,50,70,90,100\%$ of labeled data. To analyze the impact of unlabeled data, we show different versions: spamGAN-0 (no unlabeled data), spamGAN-50 (50% unlabeled data), spamGAN-70 (70% unlabeled) and spamGAN-100. Co-Train, PU-Learn results are for $50\%$ unlabeled data. We also show the performance of our base classifier (without generator, discriminator, trained on real labeled data to minimize $\mathcal{L^{(C_{\text{R}})}}$). All experiments are repeated $10$ times and the mean, standard deviation are reported.

4.0.1 Influence of Labeled Data

Table. 1 shows the classification accuracy of the different models on the test set. SpamGAN models, in general, outperform other approaches, especially when the % of labeled data is limited. When we merely use $10\%$ of labeled data, spamGAN-0, spamGAN-50, spamGAN-70, spamGAN-100 achieve an accuracy of $0.70,0.678,0.695,0.681$, respectively, which is higher than supervised approaches DRI-RCNN ($0.647$) and R-CNN ($0.538$) as well as semi-supervised approaches Co-train ($0.655$) and PU-learning ($0.508$). Even without any unlabeled data spamGAN-0 gets good results because the mutual bootstrapping between generator and classifier allows the classifier to explore beyond the small labeled training set using the fake sentences produced by the generator. The accuracy of our base classifier is $0.722$, higher than spamGAN models as GANs needs more samples to train, in general.

The accuracy of all approaches increases with % of labeled data. We select spamGAN-50 as a representative for comparison in Fig. 2. Though the difference in accuracy between spamGAN-50 and others reduces as the % of labeled data increases, spamGAN-50 still performs better than others with an accuracy of $0.856$ when all labeled data are considered.

Table. 2 shows the F1-score. We can again see that spamGAN-0, spamGAN-50 and spamGAN-70 perform better than the others, especially when the % of labeled data is small.

4.0.2 Influence of Unlabeled Data

While unlabeled data is used to augment the classifier’s performance, Fig. 3 shows that F1-score slightly decreases when the % unlabeled data increases, especially for spamGAN-100. In our case, as unlabeled data is much larger than the labeled, the generator does not entirely learn the importance of the sentence classes during pre-training (when the unlabeled sentence classes are randomly assigned), which causes problems for the classifier during adversarial training. However, when no unlabeled data is used, the generator easily learns to generate sentences conditioned on classes paving way for mutual bootstrapping between classifier and generator. We can also attribute the drop in performance to the difference in distribution of data between the unlabeled TripAdvisor reviews and the handcrafted reviews from Amazon MechanicalTurk.

4.0.3 Perplexity of Generated Sentence

We also compute the perplexity of the sentences produced by the generator (the lower the value the better). Fig. 4 shows that as the % of unlabeled data increases (spamGAN-0 to spamGAN-100), the perplexity of the sentences decreases. SpamGAN-100, SpamGAN-70 achieve a perplexity of $76.4,76.5$, respectively. Fig. 3, Fig. 4 show that using unlabeled data improves the generator in producing realistic sentences but does not fully help to differentiate between the classes which again, can be attributed to the difference in the data distribution between the labeled and unlabeled data.

Following is a sample (partial) spam sentence produced by the generator: ”Loved this hotel but i decided to the hotel in a establishment didnt look bad …the palmer house was anyplace that others said in the reviews..”. We notice that spam sentences use more conservative choice of words, focusing on adjectives, reviewer, and attributes of the hotel, while non-spam sentences speak more about the trip in general.

5 Conclusion and Future Work

We have proposed spamGAN, an approach for detecting opinion spam with limited labeled data. spamGAN, apart from detecting spam, helps to generate reviews similar to the training set. Experiments show that spamGAN outperforms state-of-the-art supervised and semi-supervised techniques when labeled data is limited. While we use TripAdvisor dataset, we plan to conduct experiments on YelpZip data (overcoming the data distribution issue of MechanicalTurk reviews). As the overall spamGAN architecture is agnostic to the implementation details of the classifier, we plan to use a more sophisticated design for classifier than a simple recurrent network.