Recognition of Advertisement Emotions with Application to Computational Advertising

TL;DR

This paper advances computational advertising by developing an affective ad dataset, comparing audiovisual and EEG-based emotion recognition methods, and demonstrating improved ad placement strategies using EEG signals.

Contribution

It introduces a new affective ad dataset, evaluates CNN and EEG features for emotion recognition, and applies EEG-based insights to enhance ad placement and user experience.

Findings

CNN features outperform handcrafted audiovisual descriptors

EEG features encode ad emotions better than content-based features

Multi-task learning yields optimal emotion recognition accuracy

Abstract

Advertisements (ads) often contain strong affective content to capture viewer attention and convey an effective message to the audience. However, most computational affect recognition (AR) approaches examine ads via the text modality, and only limited work has been devoted to decoding ad emotions from audiovisual or user cues. This work (1) compiles an affective ad dataset capable of evoking coherent emotions across users; (2) explores the efficacy of content-centric convolutional neural network (CNN) features for AR vis-\~a-vis handcrafted audio-visual descriptors; (3) examines user-centric ad AR from Electroencephalogram (EEG) responses acquired during ad-viewing, and (4) demonstrates how better affect predictions facilitate effective computational advertising as determined by a study involving 18 users. Experiments reveal that (a) CNN features outperform audiovisual descriptors for content-centric AR; (b) EEG features are able to encode ad-induced emotions better than content-based features; (c) Multi-task learning performs best among a slew of classification algorithms to achieve optimal AR, and (d) Pursuant to (b), EEG features also enable optimized ad insertion onto streamed video, as compared to content-based or manual insertion techniques in terms of ad memorability and overall user experience.

Tables

Table 1. TABLE I: Summary statistics for quadrant-wise ads.

Quadrant	Mean length (s)	Mean asl	Mean val
H asl, H val	48.16	2.17	1.02
L asl, H val	44.18	1.37	0.91
L asl, L val	60.24	1.76	-0.76
H asl, L val	64.16	3.01	-1.16

Table 2. TABLE II: Extracted features for content-centric AR. +ve class proportions (as %) for val/asl in the audio and visual modalities are specified.

Attribute	Valence/Arousal
Descriptors	Audio	Video	aud+vid (A+V)
CNN	4096D FC7 features	4096D FC7 features	8192D FC7 features
Features	from 10s SGs.	extracted from keyframes	from SGs + keyframes
		sampled every 3 seconds.	over 10s intervals.
Hanjalic [17]	Per-second sound	Per-second shot change	Concatenation of
Features	energy and pitch	frequency and motion	audio-visual features.
	statistics [17].	statistics [17].
+ve class	43.8/51.9	43.4/51.6	43.8/51.9
prop (%)

Table 3. TABLE III: Ad AR from content analysis. F1 scores are presented in the form μ ± σ plus-or-minus 𝜇 𝜎 \mu\pm\sigma .

Method	Valence			Arousal
	F1 (all)	F1 (L30)	F1 (L10)	F1 (all)	F1 (L30)	F1 (L10)
Audio FC7 + LDA	0.61$\pm$0.04	0.62$\pm$0.10	0.55$\pm$0.18	0.65$\pm$0.04	0.59$\pm$0.10	0.53$\pm$0.19
Audio FC7 + LSVM	0.60$\pm$0.04	0.60$\pm$0.09	0.55$\pm$0.19	0.63$\pm$0.04	0.57$\pm$0.09	0.50$\pm$0.18
Audio FC7 + RSVM	0.64$\pm$0.04	0.66$\pm$0.08	0.62$\pm$0.17	0.68$\pm$0.04	0.60$\pm$0.10	0.53$\pm$0.19
Video FC7 + LDA	0.69$\pm$0.02	0.79$\pm$0.08	0.77$\pm$0.13	0.63$\pm$0.03	0.58$\pm$0.10	0.57$\pm$0.18
Video FC7 + LSVM	0.69$\pm$0.02	0.74$\pm$0.08	0.70$\pm$0.15	0.62$\pm$0.02	0.57$\pm$0.09	0.52$\pm$0.17
Video FC7 + RSVM	0.72$\pm$0.02	0.79$\pm$0.07	0.74$\pm$0.15	0.67$\pm$0.02	0.62$\pm$0.10	0.58$\pm$0.19
Audio FC7 + MTL	0.85$\pm$0.02	0.83$\pm$0.10	0.78$\pm$0.20	0.78$\pm$0.03	0.62$\pm$0.14	0.45$\pm$0.16
Video FC7 + MTL	0.96$\pm$0.01	0.94$\pm$0.07	0.82$\pm$0.25	0.94$\pm$0.01	0.87$\pm$0.12	0.63$\pm$0.29
A+V FC7 + LDA	0.70$\pm$0.04	0.66$\pm$0.08	0.49$\pm$0.18	0.60$\pm$0.04	0.52$\pm$0.10	0.51$\pm$0.18
A+V FC7 + LSVM	0.71$\pm$0.04	0.66$\pm$0.07	0.49$\pm$0.19	0.56$\pm$0.04	0.49$\pm$0.10	0.47$\pm$0.19
A+V FC7 + RSVM	0.75$\pm$0.04	0.70$\pm$0.07	0.55$\pm$0.17	0.63$\pm$0.04	0.56$\pm$0.11	0.49$\pm$0.19
A+V Han + LDA	0.59$\pm$0.09	0.63$\pm$0.08	0.64$\pm$0.12	0.54$\pm$0.09	0.50$\pm$0.10	0.58$\pm$0.08
A+V Han + LSVM	0.62$\pm$0.09	0.62$\pm$0.10	0.65$\pm$0.11	0.55$\pm$0.10	0.51$\pm$0.11	0.57$\pm$0.09
A+V Han + RSVM	0.65$\pm$0.09	0.62$\pm$0.11	0.62$\pm$0.12	0.59$\pm$0.12	0.58$\pm$0.11	0.56$\pm$0.10
A+V FC7 LDA DF	0.60$\pm$0.04	0.66$\pm$0.04	0.70$\pm$0.19	0.59$\pm$0.02	0.60$\pm$0.07	0.57$\pm$0.15
A+V FC7 LSVM DF	0.65$\pm$0.02	0.66$\pm$0.04	0.65$\pm$0.08	0.60$\pm$0.04	0.63$\pm$0.10	0.53$\pm$0.13
A+V FC7 RSVM DF	0.72$\pm$0.04	0.70$\pm$0.04	0.70$\pm$0.12	0.69$\pm$0.06	0.75$\pm$0.07	0.70$\pm$0.07
A+V Han LDA DF	0.58$\pm$0.09	0.58$\pm$0.09	0.61$\pm$0.09	0.59$\pm$0.06	0.59$\pm$0.07	0.61$\pm$0.08
A+V Han LSVM DF	0.59$\pm$0.10	0.59$\pm$0.09	0.60$\pm$0.10	0.61$\pm$0.05	0.61$\pm$0.08	0.60$\pm$0.09
A+V Han RSVM DF	0.60$\pm$0.08	0.56$\pm$0.10	0.58$\pm$0.09	0.58$\pm$0.09	0.56$\pm$0.06	0.58$\pm$0.09
A+V FC7 + MTL	0.89$\pm$0.03	0.88$\pm$0.11	0.77$\pm$0.26	0.87$\pm$0.03	0.68$\pm$0.17	0.46$\pm$0.20
A+V Han + MTL	0.77$\pm$0.04	0.79$\pm$0.07	0.74$\pm$0.15	0.78$\pm$0.04	0.73$\pm$0.11	0.58$\pm$0.22

Table 4. TABLE VI: Summary of program video statistics.

Name	Scene length (s)	Manual Rating
		Valence	Arousal
coh	127$\pm$46	0.08$\pm$1.18	1.53$\pm$0.58
ipoh	110$\pm$44	0.03$\pm$1.04	1.97$\pm$0.49
friends	119$\pm$69	1.08$\pm$0.37	2.15$\pm$0.65

Full text

1 Introduction

Advertising is a pivotal industry in today’s digital world, and advertisers showcase their products and services as not only useful, but also highly worthy and rewarding. Emotions play a crucial role in conveying an effective message to viewers, and are known to mediate consumer attitudes towards brands [1, 2, 3]. Emotions are also critical for spreading public health and safety awareness, where certain personal choices are portrayed as beneficial to improving one’s quality of life, while others are portrayed as deleterious and possibly fatal. Therefore, the ability to objectively characterize advertisements (ads) in terms of their emotional content has multiple applications– e.g., inserting appropriate ads at optimal temporal points within a video stream can benefit both advertisers and consumers of video streaming websites such as YouTube [4, 5]. Subjective experience of pleasantness (valence) and emotional intensity (arousal) are important affective dimensions [6], and both influence responses to ads in distinct ways [7]. Specifically, stimulus valence and arousal are known to influence recall of images [8], movie scenes [9] and videos [4].

While mining of ad emotions is therefore beneficial, only a few works have attempted the same. This is despite the popularity of affective computing recently, and a multitude of works predicting emotions elicited by image [10, 11], speech [12], audio [13], music [14] and movie [15, 16] content. Ad affect characterization is non-trivial as with stimuli such as music and movie clips [17, 18, 14, 15] as human emotional perception is subjective. In lieu of detecting of discrete emotion categories such as joy,* sorrow* and disgust, many affect recognition (AR) works model emotions along the valence (val) and arousal (asl) dimensions [6, 19]. Overall, AR methods are broadly classified as content-centric or user-centric. Content-centric AR characterizes emotions by examining textual, audio and visual cues [17, 18]. In contrast, user-centric AR identifies elicited emotions from facial [20] or physiological [14, 15, 16, 9] measurements acquired from the user or multimedia consumer. While enabling a fine-grained examination of transient emotions, user-centered methods may nevertheless suffer from individual subjectivity.

This work expressly studies emotions conveyed by ads, and employs (i) explicit human opinions and (ii) associated content and user-centric measurements (or descriptors) which influence these opinions. Firstly, we examined the efficacy of 100 diverse, carefully curated ads to coherently evoke emotions across viewers. To this end, we examined the affective first impressions of five experts and 23 novice annotators and found that the two groups agreed considerably on the asl and val ratings. Secondly, we explored the utility of Convolutional Neural Networks (CNNs) and domain adaptation for encoding emotional audiovisual (i.e., content-based) features. As the compiled ad dataset is relatively small and insufficient for CNN training, we employed domain adaptation to transfer knowledge gained from the large-scale and annotated LIRIS-ACCEDE movie dataset [21] for decoding ad emotions. Extensive experimentation confirms that CNN descriptors outperform handcrafted audio-visual descriptors proposed in [17], with a substantial improvement observed for val recognition.

Thirdly, we performed user-centric ad AR from EEG responses compiled from annotators, and found that a three-layer CNN trained on EEG features produced state-of-the-art performance for both asl and val recognition. To our knowledge, this is the first work to perform an explicit comparison of content and user-centric methods for ad AR. In addition, we explored the utility of multi-task learning and feature/decision fusion techniques for asl and val classification. Lastly, we examined if accurate encoding of ad emotions facilitated optimized insertion of ads onto a video stream, as ads contribute to revenue generation of video hosting websites such as YouTube. A study with 18 viewers confirmed that insertion of ads identified via EEG-based emotional relevance maximized ad memorability and viewing experience while watching the ad-embedded video stream. In summary, we make the following contributions:

This is one of the few works to examine AR in ads, extending findings reported in [22, 23]. It is also the only work to characterize ad emotions in terms of explicit human opinions, and underlying (content-centric) audiovisual plus (user-centric) EEG features.

• 2.

We present a carefully curated affective dataset of 100 ads and associated affective ratings. Based on statistical analyses, we note that the ad dataset is capable of evoking coherent emotions across the expert and novice annotator groups.

• 3.

We examine the utility of CNN-based transfer learning for AR. We show that CNN features, synthesized by fine-tuning Places205 Alexnet [24] effectively captures emotional audiovisual features. Experiments show that CNN features outperform handcrafted audio-visual descriptors proposed in [17].

• 4.

We compare and contrast AR achieved with audiovisual and EEG-based CNN features. The EEG-based CNN model best encodes the asl and val attributes. Also, multi-task learning to exploit feature similarities among emotionally similar ads considerably benefits ad AR. Finally, probabilistically fusing estimates of multiple classifiers achieves superior AR than unimodal classifiers.

• 5.

We demonstrate how improved AR positively impacts ad memorability and user experience while watching an ad-embedded video stream. To our knowledge, this is one of the few works to demonstrate how improved estimation of ad asl and val scores can positively impact a computational advertising application.

The paper is organized as follows. Section 2 reviews related literature, while Section 3 overviews the compiled ad dataset and the EEG acquisition protocol. Section 4 presents the techniques adopted for content and user-centered ad AR, while Section 5 discusses AR results. Section 6 describes the user study to establish how improved emotion estimation facilitates computational advertising. Section 7 summarizes the main findings and concludes the paper.

2 Related Work

To position our work with respect to the literature and highlight its novelty, we review the related work examining (a) affect recognition (b) the impact of affective ads on consumer behavior (c) computational advertising.

2.1 Affect Recognition

Both content-centric and user-centric approaches have been proposed to infer emotions evoked by multimedia stimuli. Content-centric approaches [17, 18] predict the likely elicited emotions by examining image, audio and video-based emotion correlates [17, 25, 23]. In contrast, user-centric AR methods [14, 15, 16] estimate the stimulus-evoked emotion based on physiological changes observed in viewers (content consumers). Physiological signals indicative of emotions include pupillary dilation [26], eye-gaze patterns [9, 27] and neural activity [14, 15, 28]. Both content and user centric methods require labels denoting stimulus emotion, and such labels are compiled from annotators whose affective opinions are deemed acceptable [29, 30], given that emotion perception is highly subjective. In this work, we show that a carefully curated set of 100 ads are assigned very similar emotional labels by two independent groups comprising experts and novice annotators. Emotional attribute (i.e., asl and val) labels for these ads are then predicted via content and user-based methods. User-centered AR is achieved via EEG signals acquired with a wireless and wearable Emotiv headset, which is minimally intrusive and facilitates naturalistic user behavior.

2.2 Emotional impact of ads

Ad-induced emotions influence consumer behavior significantly [1, 2]. Work described in [3] concludes that ad-evoked feelings impact viewers explicitly as well as implicitly, and influence change in user attitudes towards (especially hedonistic) products. While many works have examined the correlation between ad emotions and user behavior, very few works have exploited these findings for developing targeted advertising mechanisms. The only work that incorporates emotional information for modeling context in advertising is CAVVA [4], where ad-in-video insertion is modeled as a discrete optimization problem based on emotion relevance between video scenes and an inventory of ads. Based on consumer psychology rules, video scenes are matched with ads with respect to asl and val scores to determine (a) the suitable ads for presentation and (b) optimal ad insertion points that would maximize user engagement.

Two recent and closely related works [23, 22] discuss how efficient affect recognition from ads via deep learning and multi-task learning lead to improved online viewing experience. This work builds on [23, 22] to show via extensive experiments that CNNs best encode emotions from both content and user-centered cues. Also, learning feature similarities among related stimuli (e.g., high asl, high val and high asl, low val clips) via multi-task learning (MTL) can benefit AR in a data impoverished setting involving only 100 labeled ad exemplars. MTL achieves excellent AR performance with both audiovisual and EEG descriptors. Finally, probabilistically fusing the estimates of multiple classifiers (decision fusion) is found to improve AR performance over unimodal classification. As a demonstration of how better affect estimation impacts a real-life application, we show how audiovisual and EEG-based CNN models enable optimized insertion of ads onto a video sequence with respect to manual performance via the CAVVA framework [4]. The user study confirms that the EEG-based CNN model, which achieves the best AR performance, also results in maximum ad memorability and the best experience for viewers watching an ad-embedded video stream.

2.3 Computational advertising

Exploiting affect recognition models for commercial applications has been a growing trend in recent years. The field of computational advertising focuses on presenting contextually relevant ads to multimedia users for commercial benefits, social good or to induce behavioral change. Despite the fact that ads are emotional, computational advertising methods have essentially matched low-level visual and semantic properties between video segments and candidate ads [31] for ad display, ignoring emotional relevance. A paradigm shift in this regard was introduced by the CAVVA framework [4], which proposed an optimization-based approach to insert ads onto video based on the emotional relevance between the video scenes and candidate ads. CAVVA employed a content-centric approach to match video scenes and ads in terms of emotional valence and arousal. However, this could be replaced by an interactive and user-centric framework as described in  [26]. We explore the use of both content-centric (via audiovisual CNN features) and user-centric (via EEG features) methods for formulating an ad-insertion strategy. A user study shows an EEG-based strategy achieves optimal user experience and also performs best with respect to ad memorability. The following section positions our work with respect to the literature.

2.4 Analysis of related work

Examination of the literature reveals that (1) AR studies are typically hampered by subjectivity in emotion perception, and a control dataset that can coherently evoke emotions in users is essential for effectively learning content or user-based emotion predictors; (2) Despite the fact that ads are emotional, and that ad emotions significantly impact user behavior, very little effort has been devoted towards incorporating emotional video-ad relevance in a computational advertising framework.

In this regard, we present the first work to compile a control set of affective ads which elicit concordant opinions from both experts and naive users. Also, we leverage CNNs for learning both audiovisual and EEG-based emotion predictors. Optimal AR is achieved with a CNN classifier employing EEG features, while CNN-based audiovisual descriptors outperform handcrafted counterparts proposed in [17]. Finally, we also demonstrate via a user study how better affect encoding facilitates ad-to-video insertion via the CAVVA mechanism [4] to enhance user viewing experience as well as ad memorability. Details pertaining to our ad dataset are presented below.

3 Advertisement Dataset

This section presents details regarding our ad dataset along with the protocol employed for collecting user ratings and EEG responses for user-centric AR.

3.1 Dataset Description

The circumplex emotion model [6] defines valence as the feeling of pleasantness/unpleasantness and arousal as the intensity of emotional feeling. Following this definition, five experts carefully compiled a dataset of 100, roughly 1-minute long commercial ads such that they were uniformly distributed over the arousal–valence plane (Figure 1). All the 100 ads are publicly available on video hosting websites, and an ad was chosen only if there was consensus among all experts on its valence and arousal labels (categorized as either high (H)/low (L)). High val ads typically involved product promotions, while low val ads were awareness messages depicting the ill effects of smoking, alcohol and drug abuse, etc. Expert labels were considered as ground-truth, and used for all recognition experiments in this paper.

We then examined if the compiled ads could serve as effective control stimuli, i.e., whether they could coherently evoke emotions across viewers. To this end, the 100 ads were independently rated by 23 annotators for val and asl upon familiarizing them with these attributes. All ads were rated on a 5-point scale, which ranged from -2 (very unpleasant) to 2 (very pleasant) for val and 0 (calm) to 4 (highly aroused) for asl. Table I presents summary statistics over the four quadrants. In our dataset, low val ads are longer and are perceived as more arousing than high val ads implying that they elicited stronger emotional reactions among viewers.

To assess whether the compiled ads evoked coherent emotions, we computed agreement among raters in terms of the (a) Krippendorff’s $\alpha$, (b) Fleiss $\kappa$ and (c) Cohen’s $\kappa$ scores. The $\alpha$ coefficient is applicable when multiple raters rate items ordinally. We obtained $\alpha=0.62$ (substantial agreement) and $0.36$ (fair agreement) respectively for val and asl, implying that valence impressions were more consistent across raters. On a coarse-grained scale, we computed the Fleiss $\kappa$ agreement among annotators. The Fleiss $\kappa$ statistic (generalization of Cohen’s $\kappa$) applies when multiple raters assign categorical values (high/low in our case) to items. Upon thresholding each rater’s asl, val scores by their mean rating to assign high/low labels for each ad, we observed a Fleiss $\kappa$ of 0.56 (moderate) for valence and 0.27 (fair) for arousal among raters. Computing Fleiss $\kappa$ upon thresholding each rater’s scores with respect to the group mean, Fleiss $\kappa$ values of 0.64 (substantial) for val and 0.30 (fair) for asl were noted. Finally, computing Cohen’s $\kappa$ agreement between each annotator and groundtruth labels (denoting expert opinion), we obtained a mean Cohen’s $\kappa$ of 0.86 (excellent agreement) and 0.68 (substantial agreement) across annotators for val and asl respectively. Overall, these observations convey that (a) greater concordance is noted among novice raters when their opinions are considered collectively rather than individually as subjectivity biases are smoothed out, (b) agreement for val is considerably higher than for asl and (c) the compiled ads evoke consistent affective impressions in the annotator and expert groups.

Another desirable property of an affective dataset is the relative independence of the asl and val dimensions [6, 32]. To examine asl-val relations for our ad dataset, we (i) examined scatter plots of the annotator ratings, and (ii) computed correlations amongst those ratings. The scatter plot of the mean asl, val annotator ratings, and the distribution of asl and val ratings are presented in Figure 1. The scatter plot is color-coded based on expert labels, and is interestingly different from the classical ‘C’ shape observed with images [33], music videos [14] and movie clips [15] attributed to the hypothesis that strong asl evokes strong val ratings. A close examination of the scatter plot reveals that a number of ads are rated as moderate asl, but high/low val. Furthermore, roughly uniform asl and val distributions are observed resulting in Gaussian fits with large variance, especially for val. This is plausible as ads are designed to convey a strong positive or negative message to viewers, while images and movie scenes may convey a relatively neutral emotion. Wilcoxon rank sum tests on ratings revealed significantly different asl ratings for high vs. low asl ads ($p<0.0001$), and distinctive val scores for high vs. low valence ads ($p<0.0001$) consistent with expectation.

Pearson correlation was computed between the asl and val dimensions with correction for multiple comparisons by limiting the false discovery rate to within 5% [34]. This procedure revealed a negative and insignificant correlation of 0.17, implying that ad asl and val scores were largely unrelated. Based on the above findings, we claim that our 100 ads constitute a control affective dataset as (i) they induce a fair range of asl and val impressions, which are also found to be largely independent; Different from the ‘C’-shape characterizing the asl-val relationship for other stimulus types, asl and val ratings are more uniformly distributed for the ad stimuli, and (ii) There is fair-to-substantial concordance among annotators in addition to considerable agreement between novice raters and the ground-truth on affective labels, implying that our ads evoked fairly coherent emotions among viewers.

3.2 EEG acquisition protocol

As annotators recorded their emotional first impressions on viewing the ads, we acquired their Electroencephalogram (EEG) brain activations via the Emotiv wireless headset. The Emotiv device comprises 14 electrodes, and has a sampling rate of 128 Hz. To maximize engagement and minimize fatigue during the rating task, these raters took a break after every 20 ads, and viewed the entire set of 100 ads over five sessions spread over two hours. Each ad was preceded by a 1s fixation cross to orient user attention, and to measure resting state EEG power used for baseline power subtraction. Upon ad viewing, the raters had a maximum of 10 seconds to input their asl and val scores via mouse clicks. Upon experiment completion, the EEG recordings were segmented into epochs, with each epoch denoting the time window corresponding to the presentation of the corresponding ad. Upon elimination of corrupted and aborted recordings, we obtained a total of 1738 epochs for 23 viewers.

Clean vs Raw EEG Data

From the recorded 1738 epochs, we manually rejected those epochs which contained head and body movement artifacts. The EEG signal was band-limited between 0.1–45 Hz, and independent component analysis (ICA) was performed to remove artifacts relating to eye movements, eye blinks and muscle movements. This process results in the removal of 212 epochs to leave us with 1526 clean epochs. Hereon, clean EEG data will refer to the 1526 preprocessed epochs after visual rejection and ICA, whereas raw EEG data will denote the original 1738 epoch data. We evaluated CNN-based AR performance on both these sets. The following section describes the content and user-centered AR techniques.

4 Content & User-centered Analysis

4.1 Content-centered Analysis

For content centered analysis, we extracted and examined audio-visual descriptors from the ads to predict the emotion (in terms of high/low asl and val) they are likely to evoke. To this end, we employed a deep convolutional neural network (CNN), and the popular handcoded audio-visual descriptors (such as motion activity, audio pitch, etc.) proposed by Hanjalic and Xu [17]. CNNs have recently become very popular for a variety of recognition problems, particularly visual [35] and audio [36], but require vast amounts of labeled training data. As our ad dataset comprised only 100 ads, we fine-tuned the pre-trained Places205 [35] model via the large-scale and labeled LIRIS-ACCEDE movie dataset [21], and employed the fine-tuned model to extract emotional descriptors for our ads. This process is termed as domain adaptation in machine learning literature.

To synthesize a deep CNN for ad AR, we employed the pre-trained Places205 model [8] originally designed for scene understanding. The Places205 CNN is trained using the Places-205 dataset comprising 2.5 million images involving 205 scene categories. The Places-205 dataset contains a wide variety of scenes captured under varying illumination, viewpoint and field of view, and we hypothesized a coherent relationship between scene perspective, lighting and the scene emotion. To find-tune the Places205 CNN, we employed the labeled LIRIS-ACCEDE dataset [21] which contains asl, val ratings for 9800 $\approx$ 10s long movie snippets. Our ads, on the other hand, are about a minute long with individual ads having lengths ranging from 30–120s.

4.1.1 FC7 Feature Extraction via CNNs

For extracting deep audio-visual features, we input to the Places205 CNN key-frame images for the visual modality, and spectrograms for audio. We fine-tuned Places205 via the LIRIS-ACCEDE [21] dataset, and employed this model to extract high-level features output by the penultimate fully connected (FC7) CNN layer.

Keyframes as Visual Descriptors

From each video in the ad and LIRIS-ACCEDE datasets, we uniformly sampled one key frame every three seconds– this enabled extraction of a continuous video profile for affect prediction. This process generated a total of 1791 key-frames for our 100 ads.

Spectrograms as Audio Descriptors

Spectrograms (SGs) shown in Figure 2 are visual representations of the audio frequency spectrum and have been successfully employed for AR from music and speech [37]. Specifically, transforming the audio content to a spectrogram image allows for audio classification to be treated as a visual recognition problem. We extracted SGs over the 10s long LIRIS-ACCEDE clips and consistently from 10s ad segments. This process generated 610 SGs for our ad dataset. Following [37], we combined multiple tracks to obtain a single spectrogram (as opposed to two for stereo). Each spectrogram is generated using a 40 ms window short time Fourier transform (STFT), with 20 ms overlap. Larger densities (denoted by red and yellow shades) of high frequencies can be noted in the spectrograms for high asl ads, and these intense scenes are often characterized by high frequency audio (e.g., sudden loud sounds). Conversely, low asl ads tend to retain a sense of continuity in the audio profile, and therefore contain high densities of low frequency sounds.

CNN Training for audio-visual features

We used the Caffe [38] deep learning framework for fine-tuning Places205 with a momentum of 0.9, weight decay of 0.0005, and a base learning rate of 0.0001 reduced by $\frac{1}{10}^{th}$ every 20000 iterations. We totally trained four binary classification networks to recognize high and low asl/val from audio/visual features. To fine-tune Places205, we used only the top and bottom 1/3rd LIRIS-ACCEDE videos in terms of asl and val rankings under the assumption that descriptors learned for these extreme-rated clips will effectively model the emotions conveyed by our ads. 4096-dimensional FC7 layer descriptors extracted from the four networks for our ads were used in the classification experiments.

4.1.2 AR with low level audio-visual features

We benchmark AR performance achieved with CNN features against the handcrafted features proposed by Hanjalic and Xu [17]. Even after a decade, their work remains one of the most popular AR baselines as seen from recent works such as [14, 15]. In [17], asl and val are modeled via low-level descriptors describing motion activity, colorfulness, shot change frequency, voice pitch and sound energy in the scene. These predictors are intuitive and interpretable, and are used to estimate time-continuous asl and val levels in [17]. Table II summarizes audiovisual features for content-centric AR, and the proportion of positive class samples for val and asl available with audio and video modalities. We attempt asl/val classification at the keyframe/spectrogram level, and class probabilities are aggregated to obtain ad-level scores for the application presented in Sec. 6.

4.2 User-centered analysis

The 1738 epochs obtained from the EEG acquisition process were used for user-centered analysis. However, these epochs were of different lengths as ad durations were variable. To maintain dimensional consistency, we performed user-centric AR experiments with (a) the first 3667 samples ($\approx 30s$ of EEG data), (b) the last 3667 samples and (c) the last 1280 samples (10s of EEG data) from each epoch(consistent with content-centered analysis, and to examine temporal effects on AR). Each epoch sample comprised data from 14 EEG channels, and all epoch samples were input to a classifier upon vectorization. In addition to conventional classifiers, we also used a deep neural network to classify EEG epochs whose architecture is described below.

4.2.1 EEG Feature Extraction for CNN Training

As we had a relatively small number of epochs (1738) with very high dimensionality (14 channels $\times$ 3667 time points = 51338 dimensional vector), a CNN trained on this data is highly susceptible to overfitting. To alleviate overfitting, we applied Principal Component Analysis (PCA) on the vectorized epochs to reduce dimensionality. PCA has been successfully employed for CNN-based EEG classification recently [39, 40], and a number of works have stressed the need for PCA-based pre-processing for robust EEG signal representation prior to neural network training [41, 42, 43, 44, 45]. Specifically, [39] discusses PCA effectiveness for deriving a good EEG input representation for CNNs.

CNN Training for EEG features

The dimensionality-reduced EEG features (preserving 90% data variance) were then passed to a CNN for val, asl recognition. We used a CNN architecture employed for time-series sensor data classification [46] and implemented with the Keras [47] library. The network is three layers deep with two 1-D convolutional layers followed by a fully connected layer. Training was performed with 64 $1\times 3$ filters in the 1-D convolutional layers and 128 nodes in the fully connected layer. We set a momentum factor of 0.9, weight decay of 0.0005 and a base learning rate of 0.0001. A dropout level of 0.5 was used to prevent overfitting. The model was trained for a maximum of 100 epochs, and early stopping was forced in case the validation loss increased over five successive training iterations. For both content and user-centric analysis, 80% of the compiled dataset was used for training and the remaining 20% for testing with the process repeated 10 times (10 $\times$ 5-fold cross validation).

5 Experiments and Results

We first describe classifiers and settings employed for binary content and user-centric AR, where the objective is to assign a binary (H/L) label for asl and val evoked by each ad, using the extracted fc7/low-level audiovisual/EEG features. Ad labels are provided by experts, whose opinions greatly agreed with naive users (Sec 3.1).

Classifiers

We considered both shallow and deep classifiers for content and user-centered AR. Among shallow classifiers, we employed linear discriminant analysis (LDA), linear SVM (LSVM) and radial basis function SVM (RSVM). LDA and LSVM partition training data via a separating hyperplane, while RSVM transforms input data onto a high-dimensional feature space where the positive and negative class samples can be linearly separated. FC7 features learned from audiovisual descriptors (Sections 4.1.1 and 4.2.1) were input to shallow classifiers for content-centered analysis, while EEG descriptors were fed to both shallow classifiers and the three layer CNN for user-centered AR.

In addition to the above single-task learning methods which do not exploit the underlying structure of the input data, we also explored the use of multi-task learning (MTL) for AR. When posed with the learning of multiple related tasks, MTL seeks to jointly learn a set of task-specific classifiers on modeling task relationships, which is highly beneficial when learning with few examples. Among the MTL methods available as part of the MALSAR package [48], we employed the sparse graph-regularized MTL (SR-MTL) where a-priori knowledge regarding task-relatedness is modeled in the form of a graph $R$. Given tasks $t=1...T$, with $X_{t}$ denoting training data for task $t$ and $Y_{t}$ their labels, SR-MTL jointly learns a weight matrix $W=[W_{1}\ldots W_{T}]$ such that the objective function $\sum_{t=1}^{T}\|W_{t}^{T}X_{t}-Y_{t}\|_{F}^{2}+\alpha\|WR\|^{2}_{F}+\beta\|W\|_{1}+\gamma\|W\|^{2}_{F}$ is minimized. Here, $\alpha,\beta,\gamma$ are regularization parameters, while $\|.\|_{F}$ and $\|.\|_{1}$ denote matrix Frobenius ($\ell_{2}$) and $\ell_{1}$-norms.

MTL is particularly suited for dimensional AR, and one can expect similarities in terms of audio-visual content among high val or high asl ads. We exploit underlying similarities by modeling each asl-val quadrant as a task (i.e., all H asl, H val ads will have identical task labels). Also, quadrants with same asl/val labels are deemed as related tasks, while those with dissimilar labels are considered unrelated. Task relatedness is then modeled via edge weights $\gamma_{ij}$ for the graph $R$, i.e., $\gamma_{ij}=1$ for related tasks, and $\gamma_{ij}=0$ for unrelated tasks, where ${i,j}\in 1\ldots T,i\neq j$.

The graph $R$ then guides the learning of $W_{t}$’s as shown in the three examples in Fig.3, where SR-MTL is fed with the specified features computed over the final 30 s of all ads. Darker shades denote salient MTL weights. Shot change frequency is found to be a key predictor of asl [17], and one can notice salient weights for H asl, H val ads in particular. The attributable reason is that our H asl H val ads involve frequent shot changes to maintain emotional intensity, while the mood of our H asl, L val ads is strongly influenced by semantics (depicting topics like drug and alcohol abuse, and overspeeding). Likewise, pitch amplitude is a key val predictor, and salient weights can be consistently seen over the 30s temporal window for HV ads. Finally, more salient weights for H val ads with motion activity reveals that our positive val ads involve accentuated motion.

For content-centric AR, apart from unimodal (audio (A) or visual (V)) fc7 features, we also employed feature fusion (A+V entries in Table V). Probabilistic decision fusion of the unimodal classifier outputs was attempted with audiovisual features (A$+$V DF entries in Table V), and with audiovisual and EEG (Audiovisual $+$ EEG DF in Table V) features. Audiovisual feature fusion (A$+$V) involved concatenation of fc7 A and V features over 10s windows (see Table II), while the $W_{est}$ technique [49] was employed for decision fusion (DF). In DF, the test label is assigned the index $j$, $j\in$ {H(1) ,L(0)}, corresponding to maximum class probability $P_{j}=\sum_{i=1}^{2}\alpha_{i}^{*}t_{i}p_{i}$, where $i$ denotes the constituent modalities, $p_{i}$’s denote classifier posteriors and $\{\alpha_{i}^{*}\}$ are the optimal weights maximizing test F1-score determined via a 2D grid search. If $F_{i}$ denotes the training F1-score for the $i^{th}$ modality, then $t_{i}=\alpha_{i}F_{i}/\sum_{i=1}^{2}\alpha_{i}F_{i}$ for given $\alpha_{i}$.

Metrics and Experimental Settings

We used the F1-score (F1) defined as the harmonic mean of precision and recall for evaluation. F1-score is appropriate for our setting due to the imbalance in the +ve and -ve class proportions. We compare our audiovisual fc7 and EEG features against the baseline features of Hanjalic and Xu [17]. These hand-crafted features are interpretable, and employed to estimate time-continuous asl and val levels. As the Hanjalic (Han) algorithm [17] inherently uses audiovisual features to model asl and val, we only consider (feature and decision) fusion performance in this case. User-centered AR uses only PCA-applied EEG features (Sec. 4.2.1). AR results obtained over ten repetitions of 5-fold cross validation (CV) (50 runs) are presented. CV is used to address the overfitting problem on small datasets, and optimal SVM parameters are determined via an inner five-fold CV on the training set. To examine the temporal variance in AR performance, we present F1-scores obtained over (a) all ad frames (‘All’), (b) last 30s (L30) and (c) last 10s (L10) for content-centered AR. Similarly, AR results are presented for (a) first 30s (F30), (b) last 30s (L30) and (c) last 10s (L10) for user-centered AR. These settings were chosen as EEG sampling rate is higher than for audio/video.

5.1 Results Overview

Tables V and V respectively present content-centric and user-centric AR results for the various settings described above, whereas Table V presents results on fusing the audiovisual and EEG-based classifier outputs. The highest F1 score achieved for a given modality across all classifiers and temporal settings is denoted in bold.

Content-centric analysis: Focusing on unimodal descriptors in Table V, we note that video fc7 features predict val (peak F1 = 0.79) considerably better than asl, while audio fc7 features encode asl (peak F1 = 0.68) slightly better than val (peak F1 = 0.66). Also, much superior AR is achieved with MTL (peak F1 = 0.96 for val, 0.94 for asl) as compared to single task classifiers. With single and multi-task classifiers, consistently higher F1-scores are noted with video fc7 features, implying that better emotion predictors are learned from the raw video data as compared to spectrograms.

Concerning multimodal methods, we firstly note that multimodal approaches achieve comparable or better F1 scores as compared to unimodal ones. For val, the best fusion performance (F1 = 0.75 with feature fusion and RSVM classifier) is superior compared to audio-based (F1 = 0.66), but inferior compared to video-based (F1 = 0.79) recognition. Contrastingly for asl, fusion F1-score (0.75 with DF) considerably outperforms unimodal methods (0.68 with audio, and 0.67 with video). We first examine feature-fusion approaches. Comparing A$+$V fc7 vs Han features, fc7 descriptors clearly outperform Han features with both single and multi-task methods. Performance difference is prominent for val (F1 = 0.75 with fc7 vs 0.65 with Han), while comparable recognition is achieved with either feature for asl (F1 of 0.63 with fc7 vs 0.59 with Han).

Examining AR performance with decision fusion (DF) methods, DF (F1 = 0.75) substantially outperforms feature fusion (F1 = 0.59) for asl recognition, while underpeforming for val (F1 = 0.72 with DF and 0.75 with feature fusion). Among classifiers, RSVM produces the best F1-scores for both asl and val among single-task classifiers with both unimodal and multimodal features. This indicates that the fc7 audiovisual features may not be easily linearly separable in the respective feature spaces. Nevertheless, the linear MTL model beats all single-task methods with both fc7 and Han features. MTL F1-scores in the A+V FC7 + MTL condition are considerably higher than single-task multimodal F1-scores, and the trend repeats with unimodal features as well. These observations suggest that learning underlying feature similarities among ads with similar attributes enables better separability of H and L asl/val data.

EEG-based AR

From user-centered analysis, we mainly examine whether (a) better AR is achievable by examining user responses vis-ã-vis mining the audiovisual ad content; (b) the three-layer CNN (Section 4.2.1) better encodes emotional attributes as compared to shallow classifiers, and (c) whether the considered CNN architecture could achieve similar AR performance with clean vs noisy EEG signals, as CNNs are adept at learning target encodings from disparate data.

Observing Table V, we firstly observe that EEG-based AR results are generally superior to content-centric results. The best EEG-based val and asl F1-scores are considerably higher than the best content-centered unimodal results. As with audiovisual features, EEG achieves better val recognition than asl different from findings reported in [16, 14, 15]. In this regard, we observe that positive val is found to correlate with increased activity in the frontal lobes [50], and the Emotiv device efficiently captures frontal lobe activity despite its limited spatial resolution.

Among the shallow classifiers considered with EEG data, RSVM again performs best. The three-layer CNN however outperforms shallow classifiers by far. Also, while very comparable results are achieved with the raw (noisy) and cleaned EEG data for val and asl employing shallow classifiers, larger performance differences are noted with the CNN and MTL methods. This implies that that while all approaches are able to discriminate the noisy EEG features, CNN and MTL are able to discriminate better from the cleaned data. As with audiovisual descriptors, highest F1-scores with EEG features (close to ceiling performance for both val and asl) are also obtained with MTL, reinforcing its utility for emotion recognition.

General Observations: Relatively small $\sigma$ values are observed in the ‘All’ condition with both audiovisual and EEG-based CNN features for the five-fold CV procedure in Tables V and V. These trends suggest that the corresponding classification models do not overfit. Examining temporal windows considered for audiovisual AR, significantly higher $\sigma$’s are nevertheless noted with Han features as well as with the L30 and L10 temporal segments, conveying that the corresponding models do not generalize well. Higher $\sigma$’s observed for the L30 and L10 conditions reveal considerable variance in AR performance on the terminal ad frames. Contrastingly, very similar $\sigma$’s are noted for the different temporal windows considered with EEG data in Table V.

Interestingly in Table V, one can note a considerable decrease in asl F1 scores for the L30 and L10 conditions with audio and visual features, while val F1-scores are similar to the ’All’ condition. Also, a sharp degradation in MTL performance is noted in the L30 and L10 conditions. Corresponding inferences are tabulated as follows. (1) Greater differences between ads towards endings are characterized by the large F1 variance in the L30 and L10 conditions with unimodal and multimodal features; conversely, similar AR performance is noted with EEG features for the different temporal segments. This implies that while the audiovisual information conveying ad emotion may significantly vary over time, human viewers typically tend to grasp the conveyed emotion rather instantaneously; (2) Fusion models synthesized with Han features are most prone to overfitting, given the generally larger $\sigma$ values seen with respect to other models. (3) Lower asl F1 scores in the L30 and L10 conditions highlight the limitation of using a single asl/val label (as opposed to dynamic labeling) over time. Generally lower F1-scores achieved for asl with all methods in Table V suggests that asl is more difficult to characterize than val (this could possibly explain the lower agreement for asl in Section 3.1), while coherency between val features and labels remains sustained over time.

Fusion of Content and User-Centric Modalities

Given the difference in AR performance observed on mining the content and user-centered descriptors (especially with respect to variance across temporal segments), one could possibly conclude that the audiovisual and EEG modalities encode complementary information. Therefore, we examined if probabilistic fusion of the content (A$+$V fc7) and EEG-based classifier outputs resulted in better asl/val recognition. Corresponding results are tabulated in Table V.

Comparing Table V against Tables V and V clearly reveals that fusing complementary information is beneficial. Fusion-based asl and val F1-scores are consistently better than individual counterparts, and more superior when shallow classifiers are employed to perform individual predictions (rows 1 and 3). These findings reveal the potential for fusion of content and user-centric cues as in [51, 52, 53].

6 Computational Advertising - User Study

Presented results clearly reveal that the compiled fc7 audiovisual and EEG desciptors outperform the baseline Han features for ad AR. We now demonstrate that improved AR positively impacts a computational advertising application– specifically, we show that better AR facilitates optimized insertion of ads onto streamed (e.g., YouTube) video. We characterize optimized ad insertion in terms of twin (possibly conflicting) objectives: (1) maximizing ad impact (measured in terms of ad memorability), and (2) minimially disrupting (or ideally enhancing) the viewing experience.

The research question that we seek to study here is Whether better affect estimation, as achieved by the CNN frameworks harnessing audiovisual and EEG descriptors, leads to optimal insertion of ads at appropriate scene transition points in a video sequence? A principled methodology to insert ads in video is proposed by the CAVVA algorithm [4]. CAVVA is a genetic algorithm-based optimization framework for inserting ads onto streamed video. On top of low-level feature based contextual matching as proposed by frameworks such as VideoSense [31], CAVVA models affective relevance between scenes in a video sequence and ads in an inventory to determine the (a) suitable ads to insert, and (b) the best temporal positions in the video sequence where the chosen ads should be inserted.

Based on consumer psychology insights, CAVVA proposes ad insertion rules that seek to strike a balance between (a) maximizing ad impact in terms of brand memorability (i.e., maximizing ad recall), and (b) minimally disrupting (or even enhancing) viewer engagement and experience. To examine the above research question, we performed a study with 18 users to compare ad recall and subjective quality of advertising schedules generated with affective scores estimated via (a) the content-centric audiovisual CNN model, (b) the user-centric EEG CNN model and (c) first impression ratings provided by experts. Details of the (i) ad and video datasets employed, (ii) employed ad insertion strategies and (iii) user study and associated results are as follows.

6.1 Ad and Video Datasets

For the user study, we used 28 ads (from the original 100), and three program videos. The ads were equally distributed among the four quadrants of the asl-val plane based on expert labels. The program videos were scenes from a television sitcom (friends) and two movies (ipoh and coh), which predominantly comprised social themes and situations capable of invoking high-to-low valence and moderate arousal (see Table VI for statistics). Each program video comprised eight scenes implying that there were seven possible ad-insertion points corresponding to scene transitions. The average scene length in the program videos was 118 seconds.

6.2 Advertisement insertion strategy

We used three affect estimation models (audiovisual CNN, EEG CNN and manual) to provide asl, val scores for the ads and video scenes. Asl, val scores for the 24 program video scenes (8 scenes $\times$ 3 videos) were computed as mean of the ratings (in the range [-2,2] for val and [0,4] for asl) acquired from three experts, and then rescaled to [0,1] via min-max normalization. The ad affective scores were computed as follows. For the content-centric method, we used normalized softmax class probabilities output by the video-based CNN model [54] for val estimation, and corresponding probabilities from the audio CNN for asl estimation. The mean score computed over all video/audio frames was used to the denote affective score of an ad in this method. Similarly, mean of the normalized softmax class probabilities over all EEG epochs for an ad was used to denote asl, val score via the user-centric EEG method. The average of continuous val and asl ratings in [0,1] annotated via FeelTrace [55] by five experts was used for Manual scores.

We then adopted the CAVVA framework [4] to generate nine unique video program sequences (VPSs with average length of 19.6 minutes) with ads inserted. These VPSs represent the different combinations of the three program videos and the affect estimation approach (audiovisual/EEG/manual). Exactly five (out of seven possible) ads were inserted onto each program video. 21 of the 28 chosen ads were inserted at least once into the nine video programs, with maximum and mean insertion frequencies of 5 and 2.14 respectively. Among the 21 inserted ads, 13 had been labeled as high val by experts, while 10 were labeled as high asl.

6.3 Experiment and Questionnaire Design

To evaluate the generated VPSs and thereby the efficacy of the affect estimation techniques for optimal ad insertions, we recruited 18 university undergraduates/graduates (7 female, mean age 20.1 years). Each user viewed three VPSs in random order such that each of the three VPSs were generated via a unique affect estimation approach. We used a randomized 3$\times$3 Latin square design in order to cover all the nine VPSs with every three users. Thus, each VPS was seen by six of the 18 viewers, and we have a total of 54 unique user responses (18 users $\times$ three video modes per user).

We designed the user evaluation so as to reveal whether the generated VPSs (a) included seamless ad insertions, (b) facilitated user engagement towards the VPS content and (c) ensured a pleasant overall viewing experience and maximized ad memorability (both immediate and long-term).

Recall evaluation is intended to verify if the inserted ads were attended to and remembered by viewers, and the immediate and day-after recall were objective measures quantifying the impact of ad insertion on short-term (immediate) and long-term (day-after) memorability of the VPS-embedded ads. Specifically, we measured the proportion of (i) inserted ads that were correctly recalled (Correct recall or hit rate), (ii) inserted ads that were not recalled (Forgotten or miss rate, $=1-\textit{hit rate}$) and (iii) non-inserted ads incorrectly recalled as seen (Incorrect recall or false alarm). For those inserted ads which were correctly recalled, we also assessed whether viewers perceived them to be contextually (emotionally) relevant to the program content (i.e., whether the ad insertions were perceived to be appropriate or good).

Upon viewing a VPS, the viewer was provided with a representative visual frame from each of the 28 ads and a sequence-specific response sheet to test ad recall and impression concerning insertion quality. All recall and insertion quality-related responses were acquired as binary values. In addition to these objective measures, we defined a second set of subjective user experience measures, and asked users to provide ratings on a 0–4 Likert scale for the questions below with 4 implying best and 0 denoting worst. (1) Were the ads uniformly distributed over the VPS? (2) Did the inserted ads blend well with the program flow? (3) Did the inserted ads match with the surrounding scenes in terms of content and mood? (4) What was the overall viewing experience while watching each VPS? Each user filled the recall and experience-related questionnaires immediately after watching each VPS. Viewers also filled in the day-after recall questionnaire, a day after completing the experiment.

6.4 User study results

As mentioned previously, program video scenes were assigned asl, val scores manually by experts, while the content-centric CNN (denoted as ‘Content’ hereon), EEG CNN and Manual methods were employed to estimate affective scores for ads. Overall quality of the CAVVA-generated VPS is influenced by the quality of affective ratings assigned to both the video scenes and ads. In this regard, we hypothesized that better ad affect estimation would result in optimized ad insertions from the user perspective.

Firstly, we examined if there was any similarity in the ad asl and val scores estimated by the Content, EEG and Manual approaches in terms of Pearson correlations. We found that (1) there was significant and positive correlation between asl scores generated by the Manual and EEG approaches ($\rho=0.55,p<0.005$), while asl scores computed via the Manual and Content methods ($\rho=0.13,\text{n.s.})$ as well as via Content and EEG ($\rho=-0.22,\text{n.s.}$) were largely uncorrelated. A similar pattern was noted for val scores with a highly positive and significant correlation observed between Manual and EEG ($\rho=0.80,p<0.000001$), while the Content–Manual ($\rho=0.33,p=0.08$) and the Content–EEG ($\rho=0.19,\text{n.s.}$) scores showed an insignificant positive correlation. These results are indicative of the fact that neural responses, which represent an implicit manifestation of emotional perception/expression, best reflect explicit affective impressions reported by humans. It is therefore unsurprising that a large number of recent affect prediction approaches [14, 15, 16, 56] have employed neural sensing as one of the modalities incorporating emotional information.

Based on viewers’ questionnaire responses, we computed the mean proportions for correct recall, ad forgottenness, incorrect recall and good insertions immediately and a day after the experiment. Similarly, mean subjective experience scores were computed for the three VPS generation schemes. Figure 4 summarizes the response results from which we make several interesting observations.

A key measure indicative of a successful advertising strategy is high brand recall [1, 4, 26], and the immediate and day-after recall rates observed with the three ad affect estimation methods are presented in Fig. 4 (left),(middle). A surprising result observed from Fig. 4 (left) and (middle) is that ads from the content and EEG-based VPSs are better recalled (or less forgotten) than manual-based. Content-based ad insertions were best recalled both immediately and the day-after, even though recall rates for the three ad-insertion approaches were not statistically different. Given the extensive literature connecting affective attributes and memorability, we examined if any such relationships could be inferred from the user study. Overall, we found a significant and positive correlation between ad val rating and recall ($\rho=0.44,p<0.05$) consistent with prior observations  [9], in addition to the fact that about $\frac{2}{3}$rds of user-viewed ads were positive in valence.

The recall rate was much worse for the day-after condition with a high proportion of ads being forgotten. Also, the proportion of incorrectly recalled ads was minimal in both the immediate and day-after conditions. Some discernable differences were observed in the proportion of good insertion impressions for the three methods– we remark here that ad recall and viewing experience are not positively correlated (some ads may be memorable because they adversely disrupted viewing experience); however, embedding ads at optimal temporal locations can enhance both ad recall and viewing experience. Post-hoc independent and right-tailed $t$-tests revealed that the proportion of immediate ’good insertion’ impressions was marginally higher for EEG as compared to manual ($t_{34}=1.337,p=0.095$).

A number of significant differences were nevertheless observed with respect to subjective user impressions of the VPSs generated via the three methods (Fig. 4 (right)). The EEG-based ad insertion mechanism scored highest for all the considered criteria. Specifically, uniform insertion scores were marginally higher for EEG with respect to manual ($t_{34}=1.5646,p=0.063$). A one-way balanced ANOVA on ad relevance scores revealed the significant effect of the ad-insertion strategy ($p<0.05$). Post-hoc $t$-tests further revealed that EEG-based ad relevance was significantly higher than manual ($t_{34}=2.3785,p<0.05$) or content-based ($t_{34}=2.1893,p<0.05$). EEG-based VPSs were also found to have the highest viewing experience scores, and were significantly higher with respect to manual-based VPSs ($t_{34}=1.7033,p<0.05$). No differences were noted with respect to user perceptions on ads blendings.

7 Discussion and Conclusion

This paper discusses AR from ads, and demonstrates the utility of estimating ad asl and val (more) accurately via a computational advertising application. Firstly, based on expert consensus we compiled a curative set of 100 semantically and emotionally diverse ads, and its ability to evoke varied-but-coherent emotions across viewers is examined by studying affective impressions of 14 raters. Suitability of the dataset for affective studies is confirmed by (1) the uniform distribution of asl and val ratings over the rating scale with minimal inter-correlation, and (2) good-to-excellent agreement between the expert and novice rater groups as measured in terms of Cohen’s $\kappa$ scores.

We then evaluated the efficacy of content and user-centric techniques for ad AR. At the outset, we note that content and user-centered methods encode complementary emotional information. While content-centric methods examine audiovisual cues for emotion predict, they typically do not model context which is crucial for emotion elicitation. The context may induce in the viewer an emotion very different from expectation based on the content, and therefore we hypothesized that examining user cues could be more effective as evidenced by many of the recent AR approaches.

Our extensive content-centered AR experiments confirmed that: (1) The proposed fc7 audio and visual CNN descriptors better predicted val, and overall F1-scores revealed that video features were better at encoding emotions than spectrogram-based fc7 descriptors; (2) Multimodal methods achieved better AR than unimodal ones, and the (A+V) fc7 features produced substantially better results than audiovisual Han features for val; Probabilistic decision fusion achieved superior results with respect to feature fusion for asl, but inferior results for val.

On the other hand, AR experiments with user-centric EEG features revealed that (1) EEG features produced superior AR performance than audiovisual descriptors; (2) The three-layer CNN classifier outperformed shallow classifiers trained on EEG data, and (3) Very comparable F1-scores were achieved with the CNN classifier with both raw (or noisy) and clean EEG data, even though shallow classifiers performed better with the cleaned features.

The above results confirm the hypothesis that emotions are better characterized by user-centric cues, which are inherently better modulated by context [9] than content-centric ones. Furthermore, content-centric classification results observed over different temporal windows reveal that content features coherently reflect human impressions of val over time, but not of asl. There are two possible explanations to this end: (a) Multiple studies have found that user impressions of stimulus val are more stable and consistent as compared to asl; also the audiovisual content of ads designed to convey an element of surprise/shock is likely to exhibit significant changes over time. (b) Owing to these variations, the use of a single affective label over the entire ad duration may be inappropriate, especially for asl, and seeking to predict time-varying affective labels could be more appropriate. Interestingly though, EEG-based AR results (Table V) show only a minor deviation between the F30 and L30 conditions even for asl (lower F1-scores for the L10 condition can also be attributed to fewer training data) suggests that humans are able to grasp the general mood of advertisements fairly quickly.

An overview of the cumulative AR results reveals minimal model overfitting– the variation in F1 scores across the 50 runs is fairly small in the ’All’ condition for content-centric AR and over all conditions for user-centered AR. Among classifiers, RBF SVM consistently produced the best results among single-task classifiers, implying the audiovisual as well as EEG features may not be trivially linearly seperable in their respective feature spaces. However, linear multi-task learning classifier achieved close-to-ceiling performance implying that learning commonalities among similarly labeled ads facilitates better feature separability. Finally, fusing the content and user-centric results as in Table V produced better F1-scores as compared to either modality, revealing the promise of mining both the content and user for accurate emotion prediction.

We then proceeded to check if improved emotion estimation enabled optimized ad insertion for computational advertising. Based on data compiled from 18 users, we observed that video program sequences generated via audiovisual and EEG-based affective scores were more effective in terms of ad recall and eliciting a better user experience than manually generated VPSs. Ads from content-based VPSs were recalled marginally better, both immediately and the day after. EEG-based VPSs received the highest scores for the viewing experience-related attributes. Ads in EEG-based VPSs were perceived to be (a) more uniformly distributed, and (b) more emotionally matched (or relevant) to the surrounding video scenes. Finally, EEG-based VPSs were also found to produce the best viewing experience.

The surprising finding of audiovisual and EEG-based VPSs being superior to the manual VPS can be explained as follows. Audiovisual and EEG-based asl and val scores were estimated via CNN models, and deep CNNs have recently performed comparable to or better than humans in tasks such as object recognition [58] and facial expression recognition [59] due to their ability to extract fine details from data. The CAVVA optimization framework [4] comprises two components– one for selecting ad insertion points into the program video, and another for selecting the ads. Asl scores only play a role in the choice of insertion points, whereas val scores influence both components. As the EEG-based framework performs best for both asl and val recognition, it also results in the most optimal ad insertions, and consequently in the best viewing experience. Finally, humans are better at rating attributes in relative than absolute terms [60, 61], which explains why the manually acquired ad-level asl and val scores may not be accurate (even if their general trends are consistent with the EEG scores as seen from the correlations computed in Sec. 6.4).

The importance of context for conveying emotions via audiovisual media such as movies and ads makes context modeling critical for AR. Recurrent neural networks have shown promise at encoding content and user-centric data for emotion [62] and mental state [63] recognition. Likewise, paucity of large-scale labeled datasets in the ad AR domain motivates the use of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to generate synthetic data. Future work will involve exploring these frameworks for recognizing and estimating ad emotions. Another line of research would be to develop algorithms that perform real-time emotional assessment of streamed video, and perform ad insertion on the fly. We will also focus on developing effective and principled methods for computational advertising, as CAVVA is modeled on ad-hoc rules derived from consumer psychology literature.

Acknowledgment

This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centre in Singapore Funding Initiative.