Sentiment analysis based on rhetorical structure theory: Learning deep neural networks from discourse trees

TL;DR

This paper introduces a discourse-aware sentiment analysis method using rhetorical structure theory and deep neural networks, improving accuracy by leveraging discourse trees and providing interpretability.

Contribution

It presents a novel tensor-based deep neural network (Discourse-LSTM) that processes discourse trees for sentiment analysis, incorporating data augmentation techniques to enhance performance.

Findings

Outperforms traditional bag-of-words methods in sentiment accuracy

Provides interpretability by highlighting salient discourse passages

Demonstrates robustness with data augmentation algorithms

Abstract

Prominent applications of sentiment analysis are countless, covering areas such as marketing, customer service and communication. The conventional bag-of-words approach for measuring sentiment merely counts term frequencies; however, it neglects the position of the terms within the discourse. As a remedy, we develop a discourse-aware method that builds upon the discourse structure of documents. For this purpose, we utilize rhetorical structure theory to label (sub-)clauses according to their hierarchical relationships and then assign polarity scores to individual leaves. To learn from the resulting rhetorical structure, we propose a tensor-based, tree-structured deep neural network (named Discourse-LSTM) in order to process the complete discourse tree. The underlying tensors infer the salient passages of narrative materials. In addition, we suggest two algorithms for data augmentation (node reordering and artificial leaf insertion) that increase our training set and reduce overfitting. Our benchmarks demonstrate the superior performance of our approach. Moreover, our tensor structure reveals the salient text passages and thereby provides explanatory insights.

Tables

Table 1. Table 1 : Overview of the different relation types that connect elementary discourse units (Mann & Thompson, 1988 ) .

Relation type	Description
Elaboration	Satellite provides additional details about the nucleus
Joint	No specific hierarchy between EDUs
Same-unit	Links parts of one EDU to another
Background	Satellite provides information to comprehend nucleus
Attribution	Satellite contains reporting verbs or cognitive predicates for nucleus
Comparison	Refers to similarities and dissimilarities
Temporal	Describes a specific ordering in time between units
Enablement	Satellite increases the ability to perform the action in nucleus
Contrast	Describes comparability or differences
Summary	Satellite is a shorter restatement of nucleus
Condition	Realization of nucleus depends on realization of satellite
Manner-means	Satellite tends to make realization of nucleus more likely
Cause	Satellite is a reason for nucleus
Explanation	Satellite justifies information from nucleus
Evaluation	Satellite assesses nucleus
Textual-organization	Describes the composition of the document
Topic-change	Topic has changed between units
Topic-comment	One EDU annotates another

Table 2. Table 7 : Overview of model parameters and their tuning ranges.

Predictive model	Parameter	Tuning range
Random forest	Number of randomly-sampled variables	1, 2, 3, 5, 7
	Number of trees	100, 200, 500, 1000
	Maximum depth of trees	1, 5, 10, 20, 50, 100, Unconstrained
Tree-LSTM	Regularization strength	0.001, 0.01, 0.05
	Input dimension for sentiment scores	1
	Input dimension for word embeddings	50
	Memory size	10, 20, 50
	Learning rate	0.00001, 0.0001, 0.001
Discourse-LSTM	Input dimension for sentiment scores	1
	Input dimension for word embeddings	50
	Memory size	10, 20, 50
	Regularization strength	0.001, 0.01, 0.05
	Learning rate	0.00001, 0.0001, 0.001

Equations

$${\sigma }_i=\frac{1}{\left|\left\{w|w\in i\right\}\right|}\sum _{w\in i}\text{pos}(w)-\text{neg}(w),$$

$${\sigma }_i=\frac{1}{\left|\left\{w|w\in i\right\}\right|}\sum _{w\in i}{e}_i^w,$$

$${\tilde{h}}_j$$

$$i_j$$

$$f_{jk}$$

$$o_j$$

$$u_j$$

$$c_j$$

$$h_j$$

$$i_j$$

$$f_{jk}$$

$$o_j$$

$$u_j$$

$$c_j$$

$$h_j$$

$$y=\underset{\hat{y}}{\mathrm{argmax}}p(\hat{y}|h_{\mathrm{root}})=\underset{\hat{y}}{\mathrm{argmax}}\mathrm{softmax}\left(W^{(s)}{h}_{\mathrm{root}}+b^{(s)}\right),$$

$${\mathbf{W}}^{(c)}$$

$${\mathbf{W}}^{(h)}$$

$${\hat{h}}_k$$

$${\hat{c}}_k$$

$${\tilde{h}}_j$$

$$i_j$$

$$f_{jk}$$

$$o_j$$

$$u_j$$

$$c_j$$

$$h_j$$

$${\hat{h}}_k$$

$${\hat{c}}_k$$

$$i_j$$

$$f_{jk}$$

$$o_j$$

$$u_j$$

$$c_j$$

$$h_j$$

$${\sigma }_l$$

$${\sigma }_r$$

Full text

Declarations of interest: none

††journal: Expert Systems with Applications

1 Introduction

Sentiment analysis reveals personal opinions towards entities such as products, services or events, which can benefit organizations and businesses in improving their marketing, communication, production and procurement. For this purpose, sentiment analysis quantifies the positivity or negativity of subjective information in narrative materials (Pang & Lee, 2008; Feldman, 2013; Chen et al., 2017; Kratzwald et al., 2018). Among the many applications of sentiment analysis are tracking customer opinions (Tanaka, 2010; Araque et al., 2017; Bohanec et al., 2017), mining user reviews (Ye et al., 2009; Mostafa, 2013; Kontopoulos et al., 2013), trading upon financial news (Khadjeh Nassirtoussi et al., 2015; Kraus & Feuerriegel, 2017; Weng et al., 2018), detect social events (Yoo et al., 2018) and predicting sales (Yu et al., 2012; Rui et al., 2013).

Sentiment analysis traditionally utilizes bag-of-words approaches, which merely count the frequency of words (and tuples thereof) to obtain a mathematical representation of documents in matrix form (Manning & Schütze, 1999; Pang & Lee, 2008; Guzella & Caminhas, 2009; Dey et al., 2018). As such, these approaches are not capable of taking into consideration semantic relationships between sections and sentences of a document. In naïve bag-of-words models, all clauses are assigned the same level of relevance, which cannot mark certain subordinate clauses more than others for purposes of inferring the sentiment. Conversely, the objective of this paper is to develop a discourse-aware method for sentiment analysis that can recognize differences in salience between individual subordinate clauses, as well as the discriminate the relevance of sentences based on their function (e. g. whether it introduces a new fact or elaborates upon an existing one).

Let us, for instance, consider the two examples in Figure 1, which express opposite polarities. By simply counting the frequency of positive and negative words, we cannot discriminate between the texts, as both contain the same number of polarity terms. To reliably analyze the sentiment, it is essential to account for the semantic structure and the variable importance across passages. That is, we can identify the main clauses and then infer the correct tone of the examples by looking at them. Similarly, RST trees can locate relevant parts in lengthy texts. For instance, the concluding section of a newspaper article is typically relevant as it reports the opinion of the author.

Our method is based on rhetorical structure theory (RST), which incorporates the discourse structures of natural language. RST structures documents hierarchically (Mann & Thompson, 1988) by splitting the content into (sub-)clauses called elementary discourse units (EDUs). The EDUs are then connected to form a binary discourse tree. Here RST discriminates between a nucleus, which conveys primary, and satellite, which conveys ancillary information. The formalization of nucleus/satellite can be loosely thought of main and subordinate parts of a clause. The edges are further labeled according to the type of discourse – for instance, whether it is an elaboration or an argument. Hence, this method essentially derives the function of a text passage. Both concepts of the RST tree help in localizing essential information within documents. Hence, the goal of this work is to develop a novel approach that identifies salient passages in a document based on their position in the discourse tree and incorporates their importance in the form of weights when computing sentiment scores.

Previous research has demonstrated that discourse-related information can improve the performance of sentiment analysis (see Section 2 for details). The work by Taboada et al. (2008) is the first to combine rhetorical structure theory and sentiment analysis. In this work, the authors weigh adjectives in a nucleus more heavily than those in a satellite. Beyond that, one can reweigh the importance of passages based on their relation type (Hogenboom et al., 2015b) or depth (Märkle-Huß et al., 2017) in the discourse tree. Some methods prune the discourse trees at certain thresholds to yield a tree of fixed depth, e. g. $2$ or $4$ levels (Märkle-Huß et al., 2017). Other approaches train machine learning classifiers based on the relation types as input features (Hogenboom et al., 2015a). What the previous references have in common is that they try to map the tree structure onto mathematically simpler representations, thereby dropping partial information from the tree.

An alternative strategy is to apply tree-structured neural networks that traverse discourse trees for representation learning. When encountering a node, these networks combine the information from the leaves and pass them on to the next higher level, until reaching the root at which point a prediction is made. Thereby, the approach merely adheres to the tree-structure but does not account for either the relation type or whether it is a nucleus/satellite. To do so, one can extend the network to include different weights for each edge in the tree depending on, e. g., the relation type. This essentially introduces additional degrees of freedom that can weigh the different discourse units by their importance. The work by Fu et al. (2016) extends the network by such a mechanism with respect to the nucleus/satellite information but discards the relation type and merely applies the network to individual sentences instead of longer documents. The approach in Ji & Smith (2017) can only exploit the relation type and not the nucleus/satellite information. Furthermore, former approaches are based on traditional recursive neural networks, which are limited by the fact that they can persist information for only a few iterations (Bengio et al., 1994). Therefore, these methods struggle with complex discourses, while we explicitly build upon tree-shaped long short-term memory models, since they are better equipped to handle very deep structures.

We build upon the previous works and advance them by proposing a specific neural network, called Discourse-LSTM. The Discourse-LSTM utilizes multiple tensors to localize salient passages within documents by incorporating the full discourse structure including nucleus/satellite information and relation types. In brief, our approach is as follows: we utilize rhetorical structure theory to represent the semantic structure of a document in the form of a hierarchical discourse tree. We then obtain sentiment scores for each leaf by utilizing both polarity dictionaries and word embeddings. The resulting tree is subsequently traversed by the Discourse-LSTM, thereby aggregating the sentiment scores based on the discourse structure in order to compute a sentiment score for the document. This approach thus weighs the importance of (sub-)clauses based on their position and relation in the discourse tree, which is learned during the training phase. As a consequence, this allows us to enhance sentiment analysis with discourse information. Another key contribution is that we propose two techniques for data augmentation that facilitate training and yield higher predictive accuracy.

The remainder of this paper is structured as follows. Section 2 reviews discourse parsing and RST-based sentiment analysis. Section 3 then introduces our Discourse-LSTM, as well as our algorithms for data augmentation. Section 4 describes our experimental setup in order to evaluate the performance of our deep learning methods in comparison to common baselines (Section 5). Section 6 concludes with a summary and suggestions for future research.

2 Background

2.1 Rhetorical structure theory

Rhetorical structure theory formalizes the discourse in narrative materials by organizing sub-clauses, sentences and paragraphs into a hierarchy (Mann & Thompson, 1988). The premise is that a document is split into elementary discourse units, which constitute the smallest, indivisible segments. These EDUs are then connected by one of $18$ different relation types, which represent edges in the discourse tree; see Table 1 for a list. Each relation is further labeled by a hierarchy type, i. e. either as a nucleus ($N$) or a satellite ($S$). Here a nucleus denotes a more essential unit of information, while a satellite indicates a supporting or background unit of information. We note that RST also defines cases where both children are labeled as nuclei at the same time. Figure 2 presents an example of a discourse tree. Here the label elaboration at the root indicates that sentence 3 provides an additional detail about the content (i. e. the comedy) of the left sub-tree. Furthermore, background reveals that sentence 1 increases the comprehensibility of sentence 3, since it is needed to make sense of the phrase “all in all”missing.

Previous research has proposed various methods for automating the parsing of discourse trees of documents. Common implementations for documents consisting of multiple paragraphs are represented by the high-level discourse analyzer HILDA (Hernault et al., 2010) and the DPLP parser (Ji & Eisenstein, 2014), of which the DPLP parser currently achieves the better F1-score in identifying relation types. Although DPLP is slightly outperformed by HILDA in EDU span detection by $1.4\text{\,}\mathrm{\char 37\relax}$ in terms of the F1-score, it shows an improvement of $2.6\text{\,}\mathrm{\char 37\relax}$ and $6.9\text{\,}\mathrm{\char 37\relax}$ on identifying the hierarchy and relation types, respectively (Ji & Eisenstein, 2014). Since inferring relation types is regarded as the most challenging subtask of RST parsing, we decided to utilize the DPLP parser in this work.

Besides RST, other forms of semantic representations have also been devised (Abend & Rappoport, 2017). These include logical structures which put a focus on quantifications, negations and coordination, while other works involve temporal relations, inferences and textual entailment. Further frameworks are speech-act theory and natural semantic metalanguage. However, their labeling is most often not unique and applied merely at sentence level without hierarchical structures. In contrast, RST specifically entails characteristics that provide benefits in our case: we obtain a hierarchical and fully-connected representation that covers the complete document (Liu & Lapata, 2018). Accordingly, our methodology adapts to the underlying story, discriminating between key messages and supplementary materials.

2.2 Sentiment analysis with RST

Previous studies have advocated different approaches for sentiment analysis that utilize the discourse tree. In the following, we categorize these approaches into (a) weighting rules or (b) tree-structured neural networks.

The paper by Taboada et al. (2008) is the first work that explicitly utilizes rhetorical structure theory in order to extract sentiment from linguistic content. It determines the relevance of words depending on whether they appear in a nucleus or satellite. Subsequently, further works have developed different weighting rules (see LABEL:tbl:literature_weighting). These aggregate the sentiment scores of EDUs based on the tree structure (Heerschop et al., 2011; Hogenboom et al., 2015b). However, the weights are frequently pre-determined and hand-crafted. A different stream of research also considers hierarchy labels (nucleus or satellite) of the nodes and updates the weights based on these. Examples include approaches that focus on the top-split (i. e. the root node) of the discourse tree and scale the relative importance based on (hand-crafted) weights (Taboada et al., 2008; Heerschop et al., 2011; Hogenboom et al., 2015b). The underlying weights can also be optimized using logistic regression (Chenlo et al., 2014). Hierarchy labels at leaf level also facilitate a more fine-grained evaluation (Hogenboom et al., 2015b), even though the discourse tree from above is neglected. Recent research also applies a recursive weighting scheme that utilizes a scaling factor to reduce the influence of passages from lower parts of the discourse tree (Hogenboom et al., 2015b; Märkle-Huß et al., 2017). Alternatively, one can prune the discourse tree at certain thresholds in order to yield a tree of fixed depth, e. g. $2$ or $4$ levels (Märkle-Huß et al., 2017). Some works also incorporate relation types between EDUs (Heerschop et al., 2011; Chenlo et al., 2014; Hogenboom et al., 2015b) or categorize them into contrastive or non-contrastive relations, which are then weighted separately (Zirn et al., 2011). What the previous rule-based approaches have in common is that they cannot incorporate the complete tree into their analysis and, instead, need to partially discard discourse information, i. e. the links between nodes within the tree structure.

LABEL:tbl:literature_TreeLSTM provides an overview of papers utilizing tree-structured approaches. The RST tree can be traversed with a recursive neural network (Bhatia et al., 2015); however, this approach only incorporates the relation types and lacks information regarding the hierarchy type. The work by Fu et al. (2016) applies a Tree-LSTM to the discourse trees and extends this method to discriminate nucleus and satellite but at the same time neither discerning the relation type nor applying data augmentation. A similar approach traverses the RST tree with the help of a recursive neural network, while utilizing relation-specific composition matrices (Ji & Smith, 2017). However, the recursive neural network is known to struggle with complex tree structures because of vanishing or exploding gradients and, instead, we utilize a long short-term memory. Moreover, the approach sums the representations in each recursion and, hence, cannot distinguish the hierarchy, i. e. between nucleus and satellite. Hence, the objective of this paper is to extend the previous works by advancing representation learning in order to incorporate the complete discourse tree, including relation types, tree depth and hierarchy labels.

The features used by the aforementioned papers differ. On the one hand, sentiment scores for EDUs are computed from dictionaries (where words are labeled as positive or negative). In terms of dictionaries, common examples include SentiWordNet (Zirn et al., 2011; Heerschop et al., 2011; Hogenboom et al., 2015b, a; Liu & Lee, 2018), hand-crafted dictionaries (Taboada et al., 2008) or domain-specific dictionaries (Märkle-Huß et al., 2017). On the other hand, approaches utilize vector representations for the EDUs based on word embeddings (Fu et al., 2016; Ji & Smith, 2017). For reasons of comparability, we also utilize both a dictionary-based approach and word embeddings in order to compute sentiment features from the content of elementary discourse units.

2.3 Representation learning for sequential and tree data

Recent advances in deep neural networks have rendered it possible to learn representations of unstructured data such as sequences, texts or trees (Goodfellow et al., 2017). This can, for instance, be achieved by recurrent neural networks, which entail an internal architecture in the form of a directed cycle, thereby creating an internal state encoding dependent structures (Chen et al., 2017). Based on these, one can process texts of arbitrary length in sequential order, while the internal state learns the complete sequence and passes information from one word to the next. However, in practice, information only persists for a few iterations (Bengio et al., 1994). A viable remedy is provided by the long short-term memory (LSTM) network. The LSTM enhances recurrent neural networks by capturing long dependencies among input signals (Hochreiter & Schmidhuber, 1997).

Previous research has proposed a Tree-LSTM that can deal with representation learning for trees. This tree-structured LSTM network traverses trees bottom-up in order to generate representations of the underlying structure (Tai et al., 2015). The Tree-LSTM computes a representation for each parent node based on its immediate children and does so recursively until the root of the tree is reached. It thereby stacks individual LSTMs such that they reflect the tree structure from the input. However, the Tree-LSTM provides no possibility of incorporating additional information from the discourse trees, such as the relation type. The Tree-LSTM can be applied to RST trees and we thus rely upon it as a baseline. We later extend the naïve Tree-LSTM through two tensor structures that express the additional degrees of freedom. This results in a Discourse-LSTM that allows us to utilize the complete set of information encoded in discourse tree.

3 Discourse-based sentiment analysis with deep learning

This section introduces our discourse-based methodology, which infers sentiment scores from textual materials. Figure 3 illustrates the underlying framework and divides the procedure into steps for discourse parsing, computing low-level polarity features, data augmentation and prediction. The prediction phase implements either of the baselines or our proposed Discourse-LSTM.

3.1 Discourse parsing

We generate discourse trees for our datasets by utilizing the DPLP parser (Ji & Eisenstein, 2014). For sake of simplicity, we introduce the following notation. We denote the relation type of node $i$ as $\rho_{i}\in\{\mathrm{elaboration},\mathrm{argument},\ldots\}$. The complete list of relation types is given in Table 1. Furthermore, we introduce $\tau_{i}\in\{\mathrm{nucleus},\mathrm{satellite}\}$ as the hierarchy type of node $i$.

3.2 Polarity features

We follow common procedures in sentiment analysis and utilize both a pre-defined dictionary that labels terms as positive or negative (Pang & Lee, 2008; Feldman, 2013), and word embeddings that represent text in multiple dimensions (Fu et al., 2016; Ji & Smith, 2017).

Sentiment dictionaries have multiple advantages, as they are domain-independent and work reliably even with few training observations. In addition, one can easily exchange the underlying dictionary for one that not only measures polarity or negativity, but is concerned with other language concepts such as subjectivity, certainty or the domain-specific tone. Our experimental results are based on the SentiWordNet 3.0 dictionary (Baccianella et al., 2010), which provides sentiment labels for $117,659$ words. Based on the sentiment labels at word level, we then proceed to compute a sentiment score $\sigma_{i}$ for each EDU $i$ via

[TABLE]

where we iterate over the words $w$ in EDU $i$, while $\text{pos}(w)$ and $\text{neg}(w)$ are the positivity and negativity scores for word $w$ according to SentiWordNet. The resulting sentiment value $\sigma_{i}$ thus represents the low-level features that later serve as input to our predictive models.

In addition, we utilize a fully neural approach by incorporating multi-dimensional word embeddings which contain considerably more information than sentiment values. In particular, we employ pre-trained $50$-dimensional word embeddings from GloVe111https://nlp.stanford.edu/projects/glove/ to represent words in each EDU. Based on the word representations in each EDU $i$, we form a high-level feature vector $\sigma_{i}$, representing the EDU, via

[TABLE]

with $e^{w}_{i}$ being the word embedding of word $w$ in EDU $i$. This approach of forming representations has been shown to work well on short texts, as is the case for RST leaves (de Boom et al., 2016).

3.3 Tree-LSTM baseline

We draw upon the Tree-LSTM as a baseline similar to Fu et al. (2016), since it is widely regarded as the status quo for tree learning (Tai et al., 2015). The Tree-LSTM takes a discourse tree as input and then processes EDU features while accounting for their position in the tree. For this purpose, it stacks individual LSTMs in the form of that tree and adapts the ideas of both a memory cell and gates from traditional LSTMs, but extends these concepts to tree structures (Tai et al., 2015). Here the underlying LSTM helps to overcome the problem of exploding gradients.

In the Tree-LSTM, each node $j$ from the discourse tree is translated into a single LSTM unit, which comprises an input gate $i_{j}$, an output gate $o_{j}$, a memory cell $c_{j}$ and hidden state $h_{j}$. In contrast to the standard LSTM, the Tree-LSTM contains not a single forget gate, but rather one forget gate $f_{jk}$ for each child $k$. This allows each parent node to recursively compute a representation from its immediate children. The input vectors to each LSTM unit are given by the hidden state $h_{k}$ and the memory cell $c_{k}$ for all children $k\in C(j)$, where $C(j)$ is the set of children of parent $j$. This layout of arranging connections renders it possible for the Tree-LSTM to pass information upward in the tree, since every node can incorporate selected information from each child-LSTM. Figure 4 details the connection between the gates in a Tree-LSTM.

Our experiments later compare the performance of two different architectures of Tree-LSTM models, namely, the child-sum and $N$-ary Tree-LSTM (Tai et al., 2015). Both are common in research, but vary in their connections between input and output gates. The former, the child-sum Tree-LSTM, sums the hidden states $h_{k}$ from the children $k\in C(j)$ in order to obtain a single input to the hidden state $\tilde{h}_{\mathrm{parent}}$ of the parent. This approach discards any information regarding the order of the children, since it uses the same weights $U^{(i)}$, $U^{(f)}$, $U^{(o)}$ and $U^{(u)}$ for all children. In contrast, the $N$-ary Tree-LSTM requires a fixed, pre-defined number of $N=\left\lvert C(j)\right\rvert$ children for each inner node. It then combines the child nodes by weighting their hidden states based on parameters $U^{(i)}_{m}$, $U^{(f)}_{km}$, $U^{(o)}_{m}$ and $U^{(u)}_{m}$ dependent on the index $m=1,\ldots,N$ of the child.

Mathematically, for input $x_{j}\in\mathbb{R}^{m}$, the child-sum Tree-LSTM transition equations are defined as

[TABLE]

where $\odot$ denotes the element-wise multiplication. Moreover, the above equations contain the weights $W^{(i)}$, $W^{(f)}$, $W^{(o)}$, $W^{(u)}$, each of dimension $n\times n$ for pre-defined memory size $n$, and $b^{(i)}$, $b^{(f)}$, $b^{(o)}$, $b^{(u)}$ of length $n$. Similarly, the $N$-ary Tree-LSTM obtains its memory cell and hidden state via

[TABLE]

In order to make sentiment predictions from the Tree-LSTM at the root node, we introduce an additional feedforward classification layer. Here we utilize a softmax classifier that predicts a class label $y$ from the hidden state $h_{\mathrm{root}}$ of the root node. The softmax layer entails further weights $W^{(s)}\in\mathbb{R}^{n\times n}$ and $b^{(s)}\in\mathbb{R}^{n}$, based on which it computes the probability $p(\hat{y}\,|\,h_{\mathrm{root}})$ of the tree belonging to class $\hat{y}$ via

[TABLE]

with the negative log-likelihood of the true class label $y$ as the cost function (Goodfellow et al., 2017).

3.4 Discourse-LSTM

The following section extends the previous Tree-LSTMs through tensor structures. The Discourse-LSTM introduces two modifications that enable us to incorporate (1) the relation type between two nodes and (2) the hierarchy type (i. e. nucleus or satellite). For this purpose, we replace the usual weight matrices in the tree-structured neural networks with a higher-dimensional representation that allows for additional degrees of freedom with respect to (1) and (2). Thereby, we yield an array of weight matrices, which is formalized and implemented via a tensor.

In order to include the relation type, we replace the global LSTM that serves all nodes with one that is dependent on the relation type $r\in\{1,\dots,n\}$. Figure 5 visualizes the idea schematically. We then select an $\text{LSTM}_{\rho_{i}}$ for each node depending on its relation type $\rho_{i}$.

We incorporate the hierarchy type $\tau_{i}$ (i. e. nucleus or satellite) by additionally weighting the cell state $c_{j}$ and the hidden state $h_{j}$ before they enter the above tensor-based LSTM. For this purpose, we introduce tensor-based weights

[TABLE]

where $\bm{W}^{(c)}$ and $\bm{W}^{(h)}$ are both of dimensions $2\times n\times n$, dependent on the input dimension $n$. We then choose the weights according to the hierarchy type $\tau_{i}$ in the tree. This allows us to additionally discriminate between the influence of nuclei and satellites.

Accordingly, the Discourse-LSTM must simultaneously optimize both the tensor-based LSTM, as well as the hierarchy-related tensors $\bm{W}^{(c)}$ and $\bm{W}^{(h)}$ based on a combined objective function. We thus rearrange them as rank-3 tensors as follows: let $\bm{W}^{(x)}[r,:]$ indicate the weight tensor for relation type $r$ and $\bm{W}^{(x)}[l,:]$ denote the weight tensor for a hierarchy type $l\in\left\{\mathrm{nucleus},\mathrm{satellite}\right\}$. On this basis, we now specify the new, updated equations for calculating the cell and hidden state. As such, the child-sum Discourse-LSTM computes

[TABLE]

Similarly, the $N$-ary Discourse-LSTM computes its representations via

[TABLE]

As a result, both the $N$-ary and child-sum Discourse-LSTM integrate the complete discourse tree into the neural network. As opposed to the works in the literature review, this approach allows us to encode both the relation type and the hierarchy type.

3.5 Training data augmentation

Deep neural networks typically feature a complex structure with thousands of weights that need to be trained, which makes them prone to overfitting. A viable remedy is to artificially increase the number of training samples in order to better tune parameters. Such approaches are common in computer vision, where one extracts different crops from the same image and later considers each as a training instance. We thus propose similar techniques for tree structures that enlarge our training set. These algorithms take a tree as input and then slightly modify its structure in each epoch of training (one full training cycle on the training set). The first variant, called node reordering, swaps sub-trees, while the second, artificial leaf insertion, randomly exchanges a leaf for a node with two new children. We thereby preserve the tree structure during node reordering, whereas, in artificial leaf insertion, we experiment with how noisy modifications to the tree structure can additionally improve representation learning.

3.5.1 Node reordering

Node reordering utilizes RST trees and rearranges the positions of inner nodes while trying to preserve the inherent structure. That is, the text passages inside the nodes must maintain their original order since the content might otherwise change its meaning or grammatical structure. Our approach thus randomly chooses an inner node $n$ and relocates it to the position of its sibling $m$ in the tree. Thereby, the position of the sibling is given by the RST structure. The sibling $m$ is then moved down the tree and becomes a child of $n$. Afterwards, the previous position of $n$ is filled by one of its former children. As a result, the order of $l$, $r$ and $m$ from left to right is unchanged. The corresponding algorithm for an inner node $n$ is sketched in Figure 6.

This approach for data augmentation tries to modify the structure slightly, thereby generating potentially different representations of the same tree. The extent of reordering depends on the level of $n$, since a reordering of a node at a higher level usually has a larger effect on the overall tree structure compared to a node at a lower level.

3.5.2 Artificial leaf insertion

Artificial leaf insertion allows us to grow larger trees. Here we make subtle but explicit modifications to the tree structure and hypothesize that, even in presence of the additional noise, this still facilitates representation learning of complex trees. The insertion of leaves into a sub-tree is depicted in Figure 7. This approach randomly picks a leaf $n$ from the tree and appends two newly created child nodes $l$ and $r$ which subsequently present the leaves, while $n$ becomes an inner node. We compute $\sigma_{l}$ and $\sigma_{r}$ by multiplying $\sigma_{n}$ with random weights $\omega\in[0,1]$ and $(1-\omega)$, i. e.

[TABLE]

These update rules thus try to keep the overall information unchanged, but distribute the values from $n$ into two separate children given a certain ratio $\omega$. We finally choose the relation type $\rho_{n}$ and the hierarchy type $\tau_{n}$ randomly.

4 Experimental setup

4.1 Datasets

We build upon earlier work and utilize three common datasets. The first consists of $2000$ movie reviews from Rotten Tomatoes (Pang & Lee, 2004), for which we perform $10$-fold cross-validation and then average the predictive performance across splits. The second dataset comprises $50,000$ reviews from the Internet Movie Database (IMDb), which are split evenly into $25,000$ reviews for training and $25,000$ for testing (Maas et al., 2011). It includes, at most, $30$ reviews for any one movie, since reviews for the same movie tend to have correlated ratings. Furthermore, the training and test sets contain a disjoint set of movies to avoid correlation based on movie-specific terms. The third dataset consists of $6400$ randomly selected food reviews from the Amazon Fine Foods dataset of which $3200$ are labeled as positive and $3200$ are labeled as negative (McAuley & Leskovec, 2013). We split the dataset into $5120$ (i. e. $80\text{\,}\mathrm{\char 37\relax}$) reviews for training and $1280$ (i. e. $20\text{\,}\mathrm{\char 37\relax}$) for testing.

All corpora are preprocessed as follows: we perform tokenization, convert all characters to lowercase, and conduct stemming. The latter maps inflected words onto a base form; e. g. “enjoyed”missing and “enjoying”missing are both reduced to “enjoy”missing (Porter, 1980).

4.2 Descriptive statistics

The resulting discourse trees exhibit the following characteristics. In the case of reviews from Rotten Tomatoes, they entail $51.09$ EDUs on average, while this number plummets to $19.79$ and $7.88$ EDUs for IMDb reviews and Amazon Food reviews, respectively. The difference stems from the nature of reviews, since Rotten Tomatoes predominantly collects reviews from known critics, while IMDb and Amazon Food reviews are user-generated (and often comprise just a few sentences). The largest discourse tree contains $154$ levels. Table 2 reports the relation types and corresponding frequencies in the corpus. The higher number of relations labeled as elaboration also has to do with the nature of reviews. Often, the critic presents a thought or argument, which is then followed by further details in support of this claim. When these passages are connected with an additional thought, the EDUs are labeled as a joint, thus explaining their overall frequency. The remaining relation types are merely used for specific purposes in the narrative.

4.3 Baselines

We construct naïve benchmarks with bag-of-words as follows. We count term frequencies and convert the numerical features into a document-term matrix. As a second baseline, we also scale the term frequencies using the term frequency-inverse document frequency approach (tf-idf), which puts stronger weights on characteristic terms (Manning & Schütze, 1999). Both feature spaces are then inserted into a random forest, since this traditional machine learning classifier can detect highly non-linear relationships but still yields satisfactory performance out-of-the-box. These benchmarks allow us to distinguish the sentiment conveyed by words from that conveyed by the discourse structure.

4.4 Training process

We optimize the proposed tree-structured models according to the following process. First, sentiment scores, as well as word embeddings, are fed as leaf node representations into the models. Second, the tree-structured models compute the root node representation which can be utilized for making the prediction through a feedforward layer. Using the prediction along with the label, we then compute the cross-entropy loss made by the model and update the weights with backpropagation.

4.5 Model evaluation

We proceed analogous to Kraus et al. (2018) in order to tune the model parameters (see appendix). In the case of the random forest baseline, we identify the optimal parameters utilizing a grid search together with $10$-fold cross-validation applied to the training set. In contrast, we optimize the deep learning architectures by taking $20\text{\,}\mathrm{\char 37\relax}$ of the training data as a validation set. After each epoch, we shuffle the observations and enlarge our training set by constructing additional samples based on our technique for data augmentation. We train our deep learning architecture with early stopping and patience set to ten epochs.

5 Results

In this section, we evaluate the performance of our Discourse-LSTM and compare it to the previous baselines. In addition, we perform statistical significance tests on the receiver operating characteristics (ROC) (DeLong et al., 1988). The evaluation provides evidence that incorporating semantic structure into the task of sentiment analysis improves the predictive performance.

5.1 Dataset 1: movie reviews from Rotten Tomatoes

Table 3 details the prediction results for the dataset featuring movie reviews from Rotten Tomatoes. The naïve benchmark with tf-idf features yields a balanced accuracy of $0.746$ and an F1-score of $0.763$. The approaches with term frequencies achieve a similar performance. Here we see no clear indication that one of the baselines is consistently superior.

The simple tree learning based on the Tree-LSTM outperforms all of the previous benchmarks. It achieves a balanced accuracy of up to $0.785$ and an F1-score of $0.787$. Nevertheless, the Tree-LSTM is surpassed by the Discourse-LSTM, which boosts the balanced accuracy to $0.800$ with an F1-score of $0.805$. This amounts to an additional improvement of $0.033$ (i. e. $4.3\text{\,}\mathrm{\char 37\relax}$) in the F1-score. Altogether, the Discourse-LSTM benefits from the discourse-related information and thus performs best overall.

Statistical significance tests on the receiver operating characteristics demonstrate that the Discourse-LSTM outperforms the Tree-LSTM to a statistically significant degree at the $1\text{\,}\mathrm{\char 37\relax}$ level. Moreover, the child-sum Discourse-LSTM with node reordering improves the predictive performance significantly at the $1\text{\,}\mathrm{\char 37\relax}$ level as compared to the child-sum Discourse-LSTM without data augmentation. However, the outcomes are not statistically significant when assessing leaf insertion.

Finally, we additionally note the following patterns: (1) there is no consistent indication that either the child-sum or $N$-ary variant is consistently superior. (2) By comparing the underlying algorithms for data augmentation, the results indicate a greater increase in predictive power from node reordering as compared to leaf insertion. This emphasizes that the larger number of training samples outweighs the additional noise from reordering. (3) The RST-based approaches also outperform models utilizing actual words as features. This suggests that a large portion of sentiment-related information is encoded in the discourse structure. (4) Utilizing pre-trained word embeddings leads to strong overfitting across all models, thereby lowering the predictive performance. This result stems from the large number of trainable parameters compared to a small number of training samples.

5.2 Dataset 2: IMDb movie reviews

Table 4 reports the predictive results for the largest of the three datasets, which is based on $50,000$ IMDb movie reviews. The random forest with tf-idf achieves a performance superior to the previous task, yielding an accuracy of $0.825$ and an F1-score of $0.823$.

Tree-structured LSTMs outperform our baseline models. For instance, the $N$-ary Tree-LSTM raises the balanced accuracy and the F1-score of the naïve baselines by $0.025$ and $0.026$, respectively. Our Discourse-LSTMs achieve a similar balanced accuracy of $0.850$ compared to simple Tree-LSTMs; however, results of the Discourse-LSTMs are more consistent. It achieves an accuracy of $0.850$ and an F1-score of $0.849$ by utilizing data augmentation. Pre-trained word embeddings push the F1-score of the $N$-ary Discourse-LSTM with data augmentation to $0.852$. Again, we find no general pattern indicating that one technique for enlarging the training set scores better than the other.

Statistical tests show that the $N$-ary Discourse-LSTM with node reordering performs significantly better than the Tree-LSTM at the $10\text{\,}\mathrm{\char 37\relax}$ level. Also, the $N$-ary Discourse-LSTM with node reordering performs significantly better at the $10\text{\,}\mathrm{\char 37\relax}$ level as compared to the $N$-ary Discourse-LSTM without data augmentation.

5.3 Dataset 3: Amazon Fine Food reviews

Table 5 lists the prediction results for the dataset featuring food reviews left by Amazon users. Regarding traditional machine learning, the random forest with tf-idf features achieves a balanced accuracy of $0.742$ and an F1-score of $0.770$.

Tree-LSTMs outperform all baselines with term frequency features.For instance, the $N$-ary Tree-LSTM leads to a balanced accuracy of $0.805$ and an F1-score of $0.787$. When exploiting all information from the RST tree, the balanced accuracy increases further to $0.813$, along with an F1-score of $0.801$. Therefore, data augmentation leveraged the balanced accuracy by $0.002$ but decreased the F1-score by $0.001$. Tree-structured models utilizing pre-trained word embeddings outperform the random forest with both tf and tf-idf features, showing a balanced accuracy of $0.771$ and an F1-score of $0.762$. However, word embeddings yield inferior performance as compared to the Tree-LSTM and Discourse-LSTM with sentiment scores.

Statistical tests on the ROC curves show that the performance of the $N$-ary Discourse-LSTM is significantly better compared to both Tree-LSTMs at the $1\text{\,}\mathrm{\char 37\relax}$ level. Although the $N$-ary Discourse LSTM benefits from node reordering, showing a higher balanced accuracy and F1-score, the improvement is not significant.

5.4 Comparison

In the following, we compare our Discourse-LSTM to the relation-specific approach in Ji & Smith (2017). In contrast to ours, it sums the representations in each recursive cell and thus cannot distinguish between nucleus and satellite. In addition, their approach utilizes a recursive neural network, which is known to suffer from vanishing or exploding gradients (Bengio et al., 1994). In response to such shortcomings, we decided to utilize a long short-term memory.

We proceed as follows in order to specifically compare their approach to ours. We leave all other parameters unchanged (i. e. identical to the previous experiments). We thus feed the networks with EDU-level features from the previous dictionary-based sentiment scores. The performance measurements indicate that the resulting predictive accuracy is inferior to the Discourse-LSTM. For the dataset from Rotten Tomatoes, their approach achieves a balanced accuracy of $0.775$ and thus represents a decline of $0.025$ (i. e. $-3.2\text{\,}\mathrm{\char 37\relax}$) compared to the best-performing child-sum Discourse-LSTM. In the case of the IMDb reviews and Amazon Fine Food reviews, their approach yields a balanced accuracy of $0.831$ and $0.803$, while the Discourse-LSTM achieves $0.850$ and $0.813$, respectively. Hence, this work results in an improvement of $0.019$ (i. e. $2.3\text{\,}\mathrm{\char 37\relax}$) and $0.010$ (i. e. $1.2\text{\,}\mathrm{\char 37\relax}$).

We additionally compare our proposed methodology for diminishing the effect of overfitting against the widely-utilized dropout technique. Dropout, in contrast to our approach of data augmentation, reduces overfitting by randomly dropping out a certain share of neurons in order to improve generalizability of the network. This prevents the neurons from co-adapting too much during training (Srivastava et al., 2014).

In order to compare dropout to node reordering and leaf insertion, we perform the following experiment utilizing the Rotten Tomatoes dataset with the $N$-ary Discourse-LSTM and the Child-sum Discourse-LSTM. While training the models, we randomly choose a certain share of weights that is set to zero. Thereby, the set of dropped-out neurons changes in each iteration of training and is defined by a dropout probability $p$, i. e. the probability of a random weight being set to zero. In this comparison, we experiment with $p$ set to $0.1$, $0.2$, $0.5$ and $0.7$. For the $N$-ary Discourse-LSTM, dropout increases the balanced accuracy by $0.015$ (i. e. $1.9\text{\,}\mathrm{\char 37\relax}$), whereas data augmentation increases the balanced accuracy by $0.01$ (i. e. $1.3\text{\,}\mathrm{\char 37\relax}$). However, for the Child-sum Discourse-LSTM, data augmentation leads to a greater improvement of $0.03$ (i. e. $3.9\text{\,}\mathrm{\char 37\relax}$) compared to the improvement of $0.02$ (i. e. $2.6\text{\,}\mathrm{\char 37\relax}$) when utilizing dropout.

In a further experiment, we combine dropout, node reordering and leaf insertion in order to examine the universal applicability of our approach. In this experiment, we see an improvement of $0.03$ (i. e. $3.9\text{\,}\mathrm{\char 37\relax}$) for the Child-sum Discourse-LSTM, which is on par with the results we obtained when utilizing data augmentation alone. Yet, for the $N$-ary Discourse-LSTM, we see a performance increase of $0.02$ (i. e. $2.6\text{\,}\mathrm{\char 37\relax}$) when utilizing dropout, node reordering and leaf insertion together. Thus, this combination outperforms models that only use a single method to avoid overfitting.

5.5 Sensitivity analysis

We now investigate the sensitivity of our models to the quality of the RST parser. Therefore, we replace a varying percentage of relation types with random noise. Finally, we evaluate the performance of a $N$-ary Discourse-LSTM with noisy data and compare it to the performance on the original trees. For this analysis, we utilize the Amazon Fine Food dataset.

Table 6 shows the results. When modifying $2\text{\,}\mathrm{\char 37\relax}$ of the relation types, we see no difference in terms of balanced accuracy, but a decrease of $0.007$ points in the F1-score. By altering $10\text{\,}\mathrm{\char 37\relax}$ of the relation types, the balanced accuracy decreases to $0.803$ with an F1-score of $0.794$. This reduction is statistically significant at the $1\text{\,}\mathrm{\char 37\relax}$ level. When modifying $20\text{\,}\mathrm{\char 37\relax}$ of the relation types, the performance decreases further to a balanced accuracy of $0.794$ and an F1-score of $0.788$.

5.6 Discussion

We now investigate the trained weights of our tensor-based mechanism inside the Discourse-LSTM. This facilitates insights into how the neural network processes the discourse and infers the sentiment from the semantic structure of textual materials. Figure 8 compares the normalized weights of the tensors $\bm{U}^{(u)}_{m}$ across different relation types $m$. The values result from using a child-sum Discourse-LSTM without data augmentation. Overall, the tensor weights between both datasets are highly correlated. For instance, the correlation coefficient between IMDb and Rotten Tomatoes stands at $0.640$, statistically significant at the $1\text{\,}\mathrm{\char 37\relax}$ level. However, we observe large differences in the relative importance across the relation types. For instance, relation types such as background and textual-organization entail only marginal importance, consistent with initial expectations. In contrast, the joint relation yields among the highest weights across both datasets.

With regard to the hierarchy-related tensors, we find a greater importance (i. e. higher weights) for nuclei as compared to satellites. For instance, the IMDb movie reviews lead to a nucleus weight of $0.738$, whereas the weight of satellites totals a mere $0.588$. This is in line with our intuition and the idea of RST: nuclei are supposed to be more essential to the writer’s purpose than satellites.

Figure 9 shows the obtained results with an illustrative example. Here we color the text according to the tensor values inside the child-sum Discourse-LSTM without data augmentation. A red text color refers to more essential pieces of information as compared to blue. In example (a), the Discourse-LSTM assigns the highest relevance to the passage “All in all, I enjoyed this comedy”missing, whereas it gives the least emphasis to “I haven’t watched a movie for a long time”missing. In example (b) from Rotten Tomatoes, the Discourse-LSTM gives highest weight to the passage “Kolya is one of the richest films i’ve seen in some time”missing. It assigns the lowest relevance to the second to fourth passage, which describe the plot of the movie. In example (c) the Discourse-LSTM gives highest weight to the passages “it will be a stinker, and to everybody’s surprise ( perhaps even the studio ) the film becomes a critical darling.”missing and “The plot is deceptively simple”missing.

The above discussion confirms that the tensors build a mechanism that learns to weight the importance of sentences based on their position and relations in the discourse tree. As a result, the Discourse-LSTM can localize the relevant parts of the document and ascertain the relative importance of sentiment scores

6 Conclusion

Deep learning for natural language predominantly builds upon sequential models such as LSTMs. While these models usually achieve a high predictive power when applied to short texts, the complexity of linguistic discourse hampers performance for longer documents. As a remedy, our paper proposes an innovative, discourse-aware approach: we first parse the semantic structure based on rhetorical structure theory, thereby mapping the document onto a discourse tree that encodes its storyline. We then apply tailored tree-structured deep neural networks with an additional tensor structure that enables us to directly learn the complete discourse tree. Each of the architectures entails more than $10,000$ parameters, empowering the models to learn highly non-linear relationships.

Our findings reveal that our Discourse-LSTM substantially outperforms the baselines. For instance, the best-performing Discourse-LSTMs achieve improvements of $4.27\text{\,}\mathrm{\char 37\relax}$ (Rotten Tomatoes), $0.60\text{\,}\mathrm{\char 37\relax}$ (IMDb) and $1.52\text{\,}\mathrm{\char 37\relax}$ (Amazon Fine Food reviews) in the F1-score as compared to using simple Tree-LSTMs. These gains are partially a result of our techniques for data augmentation, which slightly alter existing trees in order to enlarge the size of the training set. Evidently, data augmentation presents a viable option to reduce the risk of overfitting. Furthermore, the underlying tensor structure learns the relative importance of passages based on their position in the discourse tree. This facilitates insights into which discourse units convey essential pieces of information.

Appendix A Tuning ranges

Acknowledgment

The valuable contribution of Ryan Grabowski is greatly appreciated.