Preserving Intermediate Objectives: One Simple Trick to Improve Learning for Hierarchical Models

TL;DR

This paper introduces a simple yet effective method for improving hierarchical neural models by preserving intermediate objectives, demonstrated on speaker trait prediction with a 25% accuracy improvement.

Contribution

The authors propose a novel algorithm that enables backpropagation through hierarchical structures, enhancing learning in models with task hierarchies.

Findings

Achieved 25% relative error reduction on POM dataset.

Demonstrated improved performance over state-of-the-art methods.

Validated the effectiveness of preserving intermediate objectives.

Abstract

Hierarchical models are utilized in a wide variety of problems which are characterized by task hierarchies, where predictions on smaller subtasks are useful for trying to predict a final task. Typically, neural networks are first trained for the subtasks, and the predictions of these networks are subsequently used as additional features when training a model and doing inference for a final task. In this work, we focus on improving learning for such hierarchical models and demonstrate our method on the task of speaker trait prediction. Speaker trait prediction aims to computationally identify which personality traits a speaker might be perceived to have, and has been of great interest to both the Artificial Intelligence and Social Science communities. Persuasiveness prediction in particular has been of interest, as persuasive speakers have a large amount of influence on our thoughts, opinions and beliefs. In this work, we examine how leveraging the relationship between related speaker traits in a hierarchical structure can help improve our ability to predict how persuasive a speaker is. We present a novel algorithm that allows us to backpropagate through this hierarchy. This hierarchical model achieves a 25% relative error reduction in classification accuracy over current state-of-the art methods on the publicly available POM dataset.

Figures

Equations

$$L_{passion}=\sum _{i=1}^n-log(P(y_i))$$

$$L_{credibility}=\sum _{i=1}^n-log(P(y_i))$$

$$L_{persuasion}=\sum _{i=1}^n-log(P(y_i))$$

$$\partial w_{i,j}=l\cdot \frac{dl}{dw}$$

$$w_{i,j}=w_{i,j}+\partial w_{i,j}$$

$$L_{passion}^{{}^{\prime }}=\sum _{i=1}^n-log(P(y_i))$$

Full text

1 Introduction

With the advent of social media and video-sharing websites like YouTube, multimedia has transformed from a carefully curated source to a constant background steam of information that has become ubiquitous in our lives. Thus, understanding what makes certain online speakers influential to a wider audience is of considerable interest[31, 20, 30, 19]. In this work, we explore how leveraging the relationship between related speaker traits such as passion(if a speaker is perceived to convey intense emotion in their content), or credibility(if the speaker is perceived as being worthy of trust) in a task hierarchy can help improve classification performance for persuasiveness. The motivation behind choosing passion and credibility as our initial subtasks stems from Aristotle’s philosophy that the key to persuasive communication lies in ethos (credibility), pathos (passion) and logos (logical cogency) [15]. Further, significant social psychology research has analyzed and established the inter-dependencies between the perceived passion, credibility and persuasiveness or a speaker [13, 5, 29].

While there has been considerable prior work on personality traits in the traditional context of psychology [3], research towards computationally identifying and predicting these traits is limited. To that end, recent progress in the computer vision, speech, and natural language processing communities has enabled us to devise methods to computationally represent these cues from the visual, acoustic, and linguistic modalities. We attempt to combine these cues in order to predict speaker persuasiveness. Unlike prior work[26, 25], we extend our model to ternary persuasiveness classification (positive, negative, neutral) which is a much more challenging task due to the neutral class, as well as much more representative of the real world, where most videos are not just strongly persuasive and not persuasive at all.

Task hierarchies are typically modeled using the stacking ensembling technique, by training classifiers for subtasks as using their predictions as features during training and inference for a final task. However, the models for the subtasks are static, and do not participate in the training for the final task. In this work, we propose a new algorithm which allows us to backpropagate through the stack while simultaneously preserving objectives at intermediate layers of a neural network, thus enabling us to get new state-of-the-art results for ternary persuasiveness prediction. The model serves to preserve (and potentially improve) the prediction quality of intermediate speaker traits while improving the final top-level classifier. Such a model also lends itself easily to semi-supervised learning settings where little data for training the intermediate objectives is available. Although in this work, we utilize this model for speaker trait prediction specifically, it is broadly applicable to other tasks which have hierarchical structure where the final prediction depends on an intermediate prediction, such as dependency parsing with an intermediate objective for POS-tagging or facial expression recognition with intermediate objectives for Action Unit classification.

This paper is organized as follows – in Section 2, we examine related work in the field of computationally identifying speaker traits. In Section 3 we describe our proposed models and in Section 4 we describe the details of our experimental setup. Section 5 presents our results and discuss interesting insights and we conclude in Section 6.

2 Related Work

Identification and analysis of high-level speaker traits has a rich basis in social psychology. Passion and credibility, in particular, are noted to be important characteristics for eloquent speech [15]. Garsten et al. [13] discuss the importance of passion in rhetoric, and several other studies have analyzed the relationship between credibility and persuasion [5, 29]. Ekman et al. [11] explore the role face, body and speech play in judgments of personality and effect, and Kleinke et al. [16] studied gaze and how it relates to making conciliatory and demanding requests.

Several speech and language traits have also been analyzed using computational methods. Argamon et al. [2] study lexical predictors of personality from text samples. Further, Biel et al. [4] try to characterize video loggers using facial expressions and the Big-Five traits [14]. Niculae et al. [24] and Danescu et al. [10] use textual language to predict betrayal and politeness, respectively.

There has also been considerable research in leveraging multiple modalities in order to identify high-level speaker traits, like passion, credibility and persuasion. The Persuasive Opinion Multimedia (POM) dataset was introduced by Park et al. [27], where they also discuss how descriptors from each modality can be used for predicting persuasiveness. Their experiments (using an SVM) show that multimodal techniques fare better than unimodal models. To identify if a speaker is perceived as passionate and/or credible, Chatterjee et al. [7] propose an ensemble classification approach which combines two models – one that assumes inter-modality conditional independence, and one that explicitly represents the correlation between the different modalities in a lower dimensional subspace. They also use the Doc2Vec [21] representation to capture the semantic content of the text modality. Mohammadi et al. [22] predict whether a speaker is perceived as persuasive by taking into account three modalities. They also examine the effect of each modality and how persuasion ties into personality prediction. In [28], Park et al. observe that passion and credibility can help improve the results of persuasiveness prediction. Using a novel deep fusion technique, Nojavanasghari et al. [25] classify persuasiveness of the speaker. They experiment with both early fusion and late fusion for prediction, and find that the latter performs better. Siddiquie et al. [32] discuss identification of politically persuasive videos using the Rallying a Crowd dataset [8].

Maragos et al. [18] discuss various methods to integrate information from different modalities. Early fusion is defined as concatenating all unimodal features into a single aggregate multimodal descriptor, whereas approaches for late fusion learn semantic concepts from unimodal features, which are subsequently used for learning the overall objective. Chatterjee et al. [7] state that while early fusion techniques combine cues from multiple modalities, they do not explicitly model the inter-modality correlations. Deep learning in multiple modalities has been used in several applications, like speech recognition [23], retrieval [33], and predicting personality traits like persuasion [25].

We also propose a novel neural architecture which leverages information across multiple modalities and intermediate tasks to achieve better classification performance on the final task, while simultaneously also improving it’s performance on the intermediate tasks. While this work is inspired by work in multitask learning[6, 1, 9], it perhaps is most closely connected to the stacking ensembling technique where a learning algorithm utilizes the predictions of many other learning algorithms as features. The principle difference is that in our approach, the models for the subtasks are not static and learning happens continuously as we allow backpropagation through the stack. While we experiment with the final task of persuasiveness prediction, the architecture we present is generalized and can be used for any similar task setting.

3 Method

3.1 Hierarchical Model (Stacking)

We model the relationship between speaker traits in the form of a graph with dependencies. In the hierarchical model, we consider persuasiveness to be the final objective of our prediction. The hierarchical model first includes intermediate classifiers for passion and credibility and utilizes their predictions as inputs to the final classifier. The architecture is shown in Figure 1. The training algorithm is as described. This utilizes the typical stacking ensembling technique where classifiers are first trained to predict passion and credibility, and their predictions are subsequently utilized as additional features during training and inference for persuasiveness prediction.

3.2 Intermediate-Objective preserving Hierarchical Model (HIO)

In this work, we present a novel ensembling model as an alternative to stacking, which retains the advantages of stacking while also being end-to-end trainable (Figure 1). Stacking is a popular approach for tasks with dependencies, where a to-level classifier has access to not only the input features from the video but also the predictions of intermediate classifiers as input before making it’s final prediction. However, the intermediate networks are static, cannot be backpropagated through and thus do not benefit in any way from information of the final task.

Our approach seeks to allow gradient flow even through these intermediate networks, while still preserving objectives at intermediate layers. This model is attractive for a variety of tasks which have hierarchical structure. The model consists of two key stages, an imaginary forward pass through the validation data to estimate distance from the intermediate objectives and a gated weight update which prevents gradient-flow for constraint-violating updates. The model introduces a hyperparameter in the form of the Acceptable Error Rate which is the degree of error we are willing to accept from our constraint. The intuition behind this hyperparameter is that it represents the extent to which we try to preserve objectives at the intermediate layers of the neural network. In the strictest case we will set this to 1 and thus only ever update when the new weights decrease our validation loss.

The size of our dataset can also affect the value we choose for this hyperparameter. For small datasets, spurious weight updates can have a large effect on performance. We set the Acceptable Error Rate to be 1.0, forcing strict monotonicity in the validation accuracies for the individual networks. For larger datasets where we can afford to occasionally make mistakes, the Acceptable Error Rate could be treated as a hyperparameter representing a tradeoff between final-objective optimization and intermediate-objective preservation. Stacking can also be viewed as a special case of this algorithm, with the Acceptable Error Rate being set to $\infty$.

This model is also attractive from the standpoint of semi-supervised learning, wherein intermediate objectives could have significantly lesser data than the final objective. For example, in the case of Action Units for Facial Expression Recognition, Action Units are considerably harder to annotate (and in fact require training to identify), as opposed to facial expressions. Thus, with sparsely trained intermediate networks, the final objective could even be used to improve the performance of intermediate networks since our model allows backpropagation through intermediate networks.

We will now follow with an explanation of the intuition behind each of the steps in the training algorithm:

Training Algorithm Input: Dataset $D$ with training labels and their target trait scores, learning rate L , acceptable error rate $\epsilon$, passion network P, credibility network C, intermediate persuasion network Pi, and final persuasion network Pe.

Algorithm:

Train passion network P:

[TABLE]

where $P(y_{i})$ is the probability of the true class for a given sample.

This step trains a network to classify a speaker into one of three classes (very passionate, mildly passionate, not passionate), based on how passionate they are perceived to be. 2. 2.

Train credibility network C:

[TABLE]

where $P(y_{i})$ is the probability of the true class for a given sample.

This step trains a network to classify a speaker into one of three classes (very credible, mildly credible, not credible), based on how credible they are perceived to be. 3. 3.

Train intermediate persuasion network Pi with modality features from video and persuasion objective. Pop last layer after training.

This step trains a persuasion network to classify speakers based on their persuasiveness. The last layer is removed. Thus at the end of this step, we have essentially performed some pretraining and brought the weights of this smaller network to a space that is useful to predict persuasion.

At the end of this stage stage, we have a pretrained passion network P, credibility network C and intermediate persuasion network Pi. 4. 4.

Compose persuasion network Pe with passion (P) and credibility (C) networks and intermediate persuasion network (Pi), followed by a fully connected network of 2 or more layers as shown in Figure 2. 5. 5.

Train end-to-end through Pe to predict persuasion

[TABLE] 6. 6.

Backpropagation Step through all layers of the network Pe

[TABLE]

[TABLE]

We backpropagate completely through the overall persuasion network. This includes the fully-connected layers, the passion network, the credibility network and the intermediate persuasion network 7. 7.

Imaginary Forward Pass with new weights for networks P and C

[TABLE]

Similarly for credibility. We do an imaginary forward pass with the validation data for the network of each of our subtasks and compute the loss. 8. 8.

Gated Update with Acceptable Error Rate ($\epsilon$) for networks P and C

if $L^{{}^{\prime}}_{passion}\leq\epsilon\cdot L_{passion}$:

no change (keep the new weights)

else:

$W_{t+1}=W_{t}$ (revert to weights from the previous step)

Similarly for credibility.

We examine if the loss computed in the imaginary forward pass for each of the smaller subtasks with the new weights violates our constraint. If these weights do violate the constraint, we revert back to the previous weights. In this way we backpropagate through the network, only allowing weight updates that we find acceptable. This allows us to completely backpropagate through the stack while simultaneously preserving intermediate objectives. We can choose to compute this loss (and consequently the constraint) on either the training data or the validation data. In practice we find that using the validation data works best, thus we can also draw connections between our method and early stopping.

Note that the imaginary forward pass and gated update constraint applies only for the models of intermediate tasks. The intermediate models are allowed to be updated when the gate condition is satisfied.

4 Experiments

In this work, we consider the problem of persuasiveness prediction for three classes of persuasiveness - very persuasive, not persuasive and neutral. We partition our data into ten speaker-independent folds and perform cross-validation on them. We use cross-validation to account for variance across test-sets, by considering 90% as train set and 10% as test set (9 fold and 1 fold). We randomly pick one fold from the training set and use it as the validation set at each experiment. This is primarily used for early stopping, where we save the best model over training epochs as measured on the validation accuracy.

4.1 Dataset

The dataset we use is the Persuasive Opinion Multimedia (POM) dataset [26]. The corpus contains a thousand movie review videos gathered from ExpoTV.com. Each video is annotated with its persuasiveness, along with confidence, credibility, dominance, humor, and passion, amongst other speaker traits, by three human annotators. The ratings are on a Likert scale [17] of $1$ to $7$. The Pearson’s correlation coefficient between the annotations of passion and persuasion and credibility and persuasion are $0.55$ and $0.73$ respectively.

In this work, we focus on predicting persuasiveness. We consider the average rating obtained for each trait as the trait score. For our task of persuasiveness classification, we consider videos with an average rating of less than $3$ as negative or greater than $5$ as positive, and a rating between $3$ and $5$ inclusive as neutral on each of the traits.

We use the visual and acoustic feature descriptors available in POM dataset [26]. We also extract features from OpenSmile features for audio [12]. Since the audio and video features are computed for short time units, we use the mean, standard deviation, minimum, maximum, and the max-min range to represent these in a definite-sized input for each data point. For text features, we use TF-IDF from the review transcripts. After extracting the features, we use the t-test feature selection method as described in [25] to reduce the learning space to the relevant features.

4.2 Multimodal Baseline

For our multimodal baseline, we use the deep late fusion model designed by Nojavanasghari et al. [25] since it has the best performance for persuasion prediction on the POM dataset. The model trains a multilayer perceptron (MLP) for each modality and then fuses the outputs of the modalities to form the input for another MLP that performs the final persuasion classification. However, the model was originally designed for binary persuasion classification (highly positive and highly negative). In order to adapt it for our problem of ternary classification, we train the modality-specific MLPs to predict three values and use these as a vector input to the fusion MLP 111We tested our reimplementation on binary classification and could match the results reported in [25].. The cross-validation accuracy is reported in Table 4.3.2.

We use this baseline as the late fusion model for pretraining the intermediate trait networks in our hierarchical algorithms (Figure 1).

4.3 Models

We implement two hierarchical models as described in section 3.1 and 3.2.

4.3.1 Stacking

The first is a simple stacking model. The modalities are combined using a deep fusion model as described in section 4.2. We then train a passion network consists of 5 layers, each with 5 hidden units an a Relu activation, and a credibility network that has the same architecture. We also pretrain a third network to predict persuasion with features from all three modalities, pop the last layer and utilize this representation as input to a final neural network trained to predict persuasion. That is, we fuse the penultimate layer of this network with the output of the passion network, the credibility network. This forms the input of our persuasion model which consists of 3 layers with 5 hidden units each and a categorical cross-entropy loss. At test time, the prediction of the passion and credibility networks are used as features in addition to the features extracted from the different modalities.

4.3.2 Hierarchical Intermediate Objective (HIO)

The second model we implement is a hierarchical model that preserves objectives at intermediate layers of the neural network, while still allowing weight updates. In this case, these are the objectives for passion and credibility for networks trained to predict each of these traits. The passion network consists of 5 layers, each with 5 hidden units an a Relu activation. The credibility network has the same architecture. The modalities are combined using a deep fusion model as described in section 4.2. We then feed this representation as input to a neural network trained to predict persuasion. We fuse the penultimate layer of this network with the output of the passion network and the credibility network, and use this as the input representation for our final persuasion model which consists of 3 layers with 5 hidden units each and a categorical cross-entropy loss. We perform early stopping on each of the networks, storing weights at every 10th epoch.

We run this model with an Acceptable Error Rate of $\infty$ for a hierarchical model (unconstrained) and $1.0$ using the algorithm presented in section 3.3. The results of this model are presented below.

[TABLE]

In Figure 4.3.2, we observe the classification performance Hierarchical Intermediate Objective (HIO) vs. Stacking for unimodal (with text) and semi-supervised learning. We observe that the HIO model gives considerable improvements in these settings. In Table 4.3.2, we present the multimodal late fusion accuracy for all models on ternary persuasiveness classification.

4.4 Semi-Supervised Learning

As a final test of the potential of our hierarchical constrained model, we experiment with a semi-supervised learning setting, in order to see whether the system is robust to situations where the lower-level models (passion and credibility, in our case) are not pre-trained with a lot of data.

Instead of using 8 training folds, we pre-train the passion and credibility models using the late fusion technique with just 2 training folds each. We then use these models in both our hierarchical models. The results are shown in Table 4.3.2 and we notice that this setting with fewer training samples performs similarly to our experiments with the whole training set.

5 Results and Discussion

The results from our experiments are detailed in Table 4.3.2. As stated earlier, the research question we aim to answer is whether the relationship between different speaker traits can be leveraged to improve the prediction accuracy of persuasiveness.

During our experiments we observed that prediction on the POM dataset is potentially unreliable due to the small size of the test set, and performance varies highly depending on the choice of the test set.

To avoid reporting artificially high results, we choose to perform cross validation by considering each of our ten speaker-independent folds so the test set is unbiased. The cross-validation causes the accuracy to stabilize across random restarts of training.

In addition, we treat persuasiveness prediction as a ternary classification problem instead of a binary one, making the task more challenging, but also more realistic as speaker traits are not always highly polar. We experimented with undersampling and oversampling to account for the class imbalance caused by a large neutral class, but due to the small size of the POM dataset we found that this leads to slightly worse performance. This is likely because using t-test leads selecting features that are reasonably discriminative, rendering sampling is unnecessary.

We observe that in the vanilla multitask model and hierarchical multitask model, reducing all the speaker traits to sharing a single parameter space over the modalities does not benefit the prediction of persuasiveness. Although it is significantly highly in accuracy than randomly selecting a score (33% chance), it is over 10% lower in accuracy than the multimodal baseline. Further, we notice that adding passion and credibility features to the persuasiveness prediction model in the hierarchical multitask improves the cross-validation accuracy. Intuitively, these observations could mean that, even though the perception of a trait might influence another trait, the traits depend differently on each modality and cannot be predicted well with shared parameters. For example, an animated person (visual) with a bright tone (acoustic) may be perceived as passionate, but is not necessarily credible.

We observe that the hierarchical model structure lends itself to strong performance in predicting speaker persuasiveness, and that passion and credibility predictions help to significantly improve this performance. The model with intermediate-objective preserving constraints maintains strong monotonicity in performance, giving the same or better results compared to simple hierarchical models. We see an enormous improvement in accuracy over the baseline model, which signifies that predicting persuasion is benefited by using the inputs from correlated speaker traits. Further, we notice that adding the intermediate-objective-preserving constraint improves the accuracy slightly. In contrast to the multitask scenario, the models for each trait are pre-trained individually and then merged to form a single model. Although all weights are updated through backpropagation after joining the models, the parameters are not shared which allows the model to learn different dependencies on modalities for each trait.

We also notice that the model trained on just the text features performs comparably to the multimodal models in both the simple MLP case (compared to late fusion) and the hierarchical case. We hypothesize that the text features themselves are extremely discriminative of the small number of examples in the POM dataset and hence do not gain much from other modalities. However, we would like to explore the benefits of each modality in future work.

In addition, a preliminary exploration of semi-supervised learning where the model is only shown very limited passion and credibility data, yields promising results. We note that this would be particularly useful in scenarios where annotations for the intermediate properties are scarce, but the final objective is easily annotated. For example, passion and credibility are abstract concepts, but persuasiveness of a movie review can be determined by the sentiment of the review (positive or negative) and whether the reviewer convinced the annotator to watch (or not watch) the movie [26].

6 Conclusion and Future Direction

For the current work, given the basis in social science as well as the high correlation in data between the traits of passion and credibility with persuasion, we focused on the relation between these three traits. In the future, we would like to see whether we can extend this model to model other high level speaker traits, or possibly even other speaker attributes (like age, gender). This could possibly enable us to test our models on more datasets, as well as real world data like political debates or advertisements.

In addition, we believe that our model which preservers intermediate objectives will generalize well to other tasks that have a hierarchical structure such as facial expression recognition, as well as in semi-supervised settings. We would like to experiment with more data that exhibits these characteristics as well as further explore our technique’s application in semi-supervised learning.