Winning on the Merits: The Joint Effects of Content and Style on Debate Outcomes

TL;DR

This paper presents a predictive model that combines content and style features to determine debate winners, demonstrating that stronger arguments and linguistic cues significantly influence debate outcomes.

Contribution

It introduces a novel model integrating content and style to predict debate results, outperforming models based solely on linguistic features.

Findings

Model predicts debate winners with 74% accuracy

Content and style features jointly improve prediction performance

Linguistic features correlate with argument strength

Abstract

Debate and deliberation play essential roles in politics and government, but most models presume that debates are won mainly via superior style or agenda control. Ideally, however, debates would be won on the merits, as a function of which side has the stronger arguments. We propose a predictive model of debate that estimates the effects of linguistic features and the latent persuasive strengths of different topics, as well as the interactions between the two. Using a dataset of 118 Oxford-style debates, our model's combination of content (as latent topics) and style (as linguistic features) allows us to predict audience-adjudicated winners with 74% accuracy, significantly outperforming linguistic features alone (66%). Our model finds that winning sides employ stronger arguments, and allows us to identify the linguistic features associated with strong or weak arguments.

Tables

Table 1. Table 1: Debate outcome prediction results for baseline models and SVMs using the various linguistic feature categories, compared to our model that includes latent argument strengths in addition to the linguistic features. The best performing system (in bold ) is achieved by our system with topic strength as latent variables when all features are used, which significantly outperforms the baselines via bootstrap resampling test ( p < 0.05 𝑝 0.05 p<0.05 ). For the lower section, each row shows features included in addition to those in the rows above.

	SVMs	Our Model
		(w Latent
		Strength)
Baselines
Ngrams	61.0	–
Audience Feedback	56.8	–
Features (as in § 3.3)
Basic	57.6	59.3
+ Style, Semantics, Discourse	59.3	65.3
+ Sentence, Argument	62.7	69.5
+ Interaction (all features)	66.1	73.7

Table 2. Table 2: Prediction results (in accuracy) with different initialization and topic strength constraints. C3 denotes a constraint that a topic cannot be strong for both sides.

	Initialization
Constraints	Freq	AllStrong	AllStrongwin	Random
C1, C2	73.7	71.2	70.3	67.8
C1, C2, C3	72.9	73.7	69.5	68.6

Table 3. Table 3: Top 3 types of shifts. t ​ o ​ p ​ i ​ c s ​ e ​ l ​ f 𝑡 𝑜 𝑝 𝑖 subscript 𝑐 𝑠 𝑒 𝑙 𝑓 topic_{self} and t ​ o ​ p ​ i ​ c o ​ p ​ p ​ o 𝑡 𝑜 𝑝 𝑖 subscript 𝑐 𝑜 𝑝 𝑝 𝑜 topic_{oppo} are the strengths of the current topic for one side and their opponent. t ​ o ​ p ​ i ​ c s ​ e ​ l ​ f ′ 𝑡 𝑜 𝑝 𝑖 superscript subscript 𝑐 𝑠 𝑒 𝑙 𝑓 ′ topic_{self}^{\prime} and t ​ o ​ p ​ i ​ c o ​ p ​ p ​ o ′ 𝑡 𝑜 𝑝 𝑖 superscript subscript 𝑐 𝑜 𝑝 𝑝 𝑜 ′ topic_{oppo}^{\prime} are the strengths of the topic for the following arguments.

	$(topi{c}_{self},topi{c}_{oppo})\to (topi{c}_{self}^{\prime },topi{c}_{oppo}^{\prime })$	Percent
Winners	(Strong, Weak) $\to$ (Strong, Weak)	12.7%
	(Strong, Strong) $\to$ (Strong, Strong)	10.5%
	(Strong, Strong) $\to$ (Strong, Weak)	8.9%
Losers	(Weak, Strong) $\to$ (Weak, Strong)	11.9%
	(Strong, Strong) $\to$ (Strong, Strong)	9.0%
	(Strong, Weak) $\to$ (Strong, Weak)	8.8%

Table 4. Table 4: Top weighted features joint with topic strength. “ f ​ u ​ l ​ l 𝑓 𝑢 𝑙 𝑙 full ” and “ i ​ n ​ t ​ e ​ r 𝑖 𝑛 𝑡 𝑒 𝑟 inter ” indicates features that are calculated for full debates or the interactive (discussion) phase only. “FS” denotes frame semantic. Two-sided Mann-Whitney rank test is conducted on between features of wining and losing sides ( ∗ ∗ \ast : p < 0.05 𝑝 0.05 p<0.05 , ∗ ∗ \ast ∗ ∗ \ast : p < 0.01 𝑝 0.01 p<0.01 ).

Category	Topic Strength
	Strong	Weak
Basic	# “we”full	# “you”inter $\ast$
	# “they”inter	# “I”inter
	# “emotion:sadness”full	# “emotion:joy”full $\ast$
	# “emotion:disgust”full	# “emotion:trust”full $\ast$ $\ast$
		# unique nounsfull $\ast$
		# “numbers”full
		# named entitiesinter $\ast$
Style, Semantic, Discourse	# non-verb hedging full	# non-verb hedging full
	avg concretenessfull $\ast$	avg arousal scorefull $\ast$
	# formal wordsfull $\ast$	# PDTB:temporalinter $\ast$
	# FS:capabilityfull	# PDTB:contrastinter
	# FS:informationfull	# FS:certaintyfull
Sentence, Argument	Flesch Reading Easefull	Flesch Reading Easefull
	# sentiment:negativefull $\ast$	# sentiment:neutralinter $\ast$
	# questionfull	# questionfull
	# audience laughterinter $\ast$
	decayed argument countfull $\ast$
Interaction	# words addressing	if addressing
	opponent’s argumentfull	opponent’s argumentfull
	# common words with
	opponent’s argumentfull

Equations

$$\underset{\mathbf{w}}{\min }\frac{1}{2}\Vert \mathbf{w}{\Vert }^2+C\cdot \sum _{i}l(-y_i\cdot \underset{{\mathbf{h}}_{\mathbf{i}}}{\max }[\mathbf{w}\cdot \tilde{\Phi }({\mathbf{x}}_{\mathbf{i}},{\mathbf{h}}_{\mathbf{i}})])$$

Full text

1 Introduction

What determines the outcome of a debate? In an ideal setting, a debate is a mechanism for determining which side has the better arguments and for an audience to reevaluate their views in light of what they have learned. This ideal vision of debate and deliberation has taken an increasingly central role in modern theories of democracy [Habermas, 1984, Cohen, 1989, Rawls, 1997, Mansbridge, 2003]. However, empirical evidence has also led to an increasing awareness of the dangers of style and rhetoric in biasing participants towards the most skillful, charismatic, or numerous speakers [Noelle-Neumann, 1974, Sunstein, 1999].

In light of these concerns, most efforts to predict the persuasive effects of debate have focused on the linguistic features of debate speech [Katzav and Reed, 2008, Mochales and Moens, 2011, Feng and Hirst, 2011] or on simple measures of topic control [Dryzek and List, 2003, Mansbridge, 2015, Zhang et al., 2016]. In the ideal setting, however, we would wish for the winning side to win based on the strength and merits of their arguments, not based on their skillful deployment of linguistic style. Our model therefore predicts debate outcomes by modeling not just the persuasive effects of directly observable linguistic features, but also by incorporating the inherent, latent strengths of topics and issues specific to each side of a debate.

To illustrate this idea, Figure 1 shows a brief exchange from a debate about the death penalty. Although the arguments from both sides are on the same subtopic (the execution of innocents), they make their points with a variety of stylistic maneuvers, including rhetorical questions, factual numbers, and logical phrasing. Underlying these features is a shared content, the idea of the execution of innocents. Consistent with the work of  ?), this subtopic appears to inherently support one side – the side opposed to the death penalty – more strongly than the other, independent of stylistic presentation. We hypothesize that within the overall umbrella of a debate, some topics will tend to be inherently more persuasive for one side than the other, such as the execution of innocents for those opposed to the death penalty, or the gory details of a murder for those in favor of it. Strategically, debaters seek to introduce topics that are strong for them and weaker for their opponent, while also working to craft the most persuasive stylistic delivery they can. Because these stylistic features themselves vary with the inherent strength of topics, we are able to predict the latent strength of topics even in new debates on entirely different issues, allowing us to predict debate winners with greater accuracy than before.

In this paper, then, we examine the latent persuasive strength of debate topics, how they interact with linguistic styles, and how both predict debate outcomes. Although the task here is fundamentally predictive, it is motivated by the following substantive questions: How do debates persuade listeners? By merit of their content and not just their linguistic structure, can we capture the sense in which debates are an exchange of arguments that are strong or weak? How do these more superficial linguistic or stylistic features interact with the latent persuasive effects of topical content? Answering these questions is crucial for modern theories of democratic representation, although we seek only to understand how these features predict audience reactions in the context of a formal debates where substance perhaps has the best chance of overcoming pure style. We discuss in detail the relevance of our work to existing research on framing, agenda setting, debate, persuasion, and argument mining in § 6.

We develop here a joint model that simultaneously 1) infers the latent persuasive strength inherent in debate topics and how it differs between opposing sides, and 2) captures the interactive dynamics between topics of different strength and the linguistic structures with which those topics are presented. Experimental results on a dataset of 118 Oxford-style debates show that our topic-aware debate prediction model achieves an accuracy of 73.7% in predicting the winning side. This result is significantly better than a classifier trained with only linguistic features (66.1%), or using audience feedback (applause and laughter; 58.5%), and significantly outperforms previous predictive work using the same data [Zhang et al., 2016] (73% vs. 65%). This shows that the inherent persuasive effect of argument content plays a crucial role in affecting the outcome of debates.

Moreover, we find that winning sides are more likely to have used inherently strong topics (as inferred by the model) than losing sides (59.5% vs. 54.5%), a result echoed by human ratings of topics without knowing debate outcomes (44.4% vs. 30.1%). Winning sides also tend to shift discussion to topics that are strong for themselves and weak for their opponents. Finally, our model is able to identify linguistic features that are specifically associated with strong or weak topics. For instance, when speakers are using inherently stronger topics, they tend to use more first person plurals and negative emotion, whereas when using weaker arguments, they tend to use more second person pronouns and positive language. These associations are what allow us to predict topic strength, and hence debate outcomes, out of sample even for debates on entirely new issues.

2 Data Description

This study uses transcripts from Intelligence Squared U.S. (IQ2) debates.111http://intelligencesquaredus.org Each debate brings together panels of renowned experts to argue for or against a given issue before a live audience. The debates cover a range of current issues in politics and policy, and are intended to “restore civility, reasoned analysis, and constructive public discourse.” Following the Oxford style of debate, each side is given a 7-minute opening statement. The moderator then begins the discussion phase, allowing questions from the audience and panelists, followed by 2-minute closing statements.

The live audience members record their pre- and post-debate opinions as pro, con, or undecided relative to the resolution under debate. The results are shared only after the debate has concluded. According to IQ2, the winning side is the one that gains the most votes after the debate. 118 debate transcripts were collected, with pro winning 60.222**?**) also use IQ2 to study talking points and their predictive power on debate outcome. Our dataset includes theirs plus 11 additional debates (excluding one with result as tie). 84 debates had two debaters on each side, and the rest had three per side. Each debate contains about 255 speaking turns and 17,513 words on average.333The dataset can be downloaded from http://www.ccs.neu.edu/home/luwang/.

These debates are considerably more structured and moderated, and have more educated speakers and audience members, than one generally finds in public debates. As such, prediction results of our model may vary on other types of debates. Meanwhile, since we do not focus on formal logic and reasoning structure, but rather on the intrinsic persuasiveness of different topics, it may be that the results here are more general to all types of argument. Answering this question depends on subsequent work comparing debates of varying degrees of formality.

3 The Debate Prediction Model

We consider debate as a process of argument exchange. Arguments have content with inherent (or exogenously determined) persuasive effects as well as a variety of linguistic features shaping that content. We present here a debate outcome prediction model that combines directly observed linguistic features with latent persuasive effects specific to different topical content.

3.1 Problem Statement

Assume that the corpus $D$ contains $N$ debates, where $D=\{d_{i}\}_{i=1}^{N}$. Each debate $d_{i}$ comprises a sequence of arguments, denoted as $\mathbf{x_{i}}=\{x_{i,j}\}_{j=1}^{n_{i}}$, where $n_{i}$ is the number of arguments. For the present purposes, an argument is a continuous unit of text on the same topic, and may contain multiple sentences within a given turn (see Figure 1). We use $\mathbf{x_{i}^{p}}$ and $\mathbf{x_{i}^{c}}$ to denote arguments for pro and con. The outcome for debate $d_{i}$ is $y_{i}\in\{1,-1\}$, where 1 indicates pro wins and -1 indicates con wins.

We assume that each debate $d_{i}$ has a topic system, where debaters issue arguments from $K$ topics relevant to the motion ($K$ varies for different debates). Each topic has an intrinsic persuasion strength which may vary between sides (e.g. a discussion of innocent convicts may intrinsically help the anti-death-penalty side more than the pro). Thus we have a topic strength system $\mathbf{h_{i}=\{\mathbf{h_{i}^{p}},\mathbf{h_{i}^{c}}\}}$, where the strengths for $K$ topics are $\mathbf{h_{i}^{p}}=\{h_{i,k}^{p}\}_{k=1}^{K}$ for pro, and $\mathbf{h_{i}^{c}}=\{h_{i,k}^{c}\}_{k=1}^{K}$ for con. Topic strength $h_{\ast,\ast}^{\ast}$ is a categorical variable in $\{\textsc{strong},\textsc{weak}\}$.444Binary topic strength is better-suited for our proposed discriminative learning framework. In exploratory work we found that continuous-value strength under the same framework tended to be pushed towards extreme values during learning. Neither the topics nor their strength are known a priori, and thus must be inferred.

For debate $d_{i}$, we define $\Phi(\mathbf{x_{i}^{p}},\mathbf{h_{i}})$ and $\Phi(\mathbf{x_{i}^{c}},\mathbf{h_{i}})$ to be feature vectors for arguments from pro and con. We first model and characterize features for each argument and then aggregate them by side to predict the relative success of each side. Therefore, the feature vectors for a side can be formulated as the summation of feature vectors of its arguments, i.e. $\Phi(\mathbf{x_{i}^{p}},\mathbf{h_{i}})=\sum_{x_{i,j}\in\mathbf{x_{i}^{p}}}\phi(x_{i,j},\mathbf{h_{i}})$, and $\Phi(\mathbf{x_{i}^{c}},\mathbf{h_{i}})=\sum_{x_{i,j}\in\mathbf{x_{i}^{c}}}\phi(x_{i,j},\mathbf{h_{i}})$, where $\phi(x_{i,j},\mathbf{h_{i}})$ is the feature vector of argument $x_{i,j}$.555This assumes the strength of arguments is additive, though it is possible that a single extremely strong or weak argument could decide a debate, or that debates are won via “rounds” rather than in aggregate.

Each argument feature in $\phi(x_{i,j},\mathbf{h_{i}})$ combines a stylistic feature directly observed from the text with a latent strength dependent on the topic of the argument. For instance, consider an argument $x_{i,j}$ of a topic with an inferred strength of strong and which contains $3$ usages of the word “you”. Then $x_{i,j}$ has two coupled topic-aware features of the form $\phi_{M(\mathtt{feature},\mathtt{strength})}(x_{i,j},\mathbf{h_{i}})$: $\phi_{M(``\#you",``strong")}(x_{i,j},\mathbf{h_{i}})$ takes a value of 3, and $\phi_{M(``\#you",``weak")}(x_{i,j},\mathbf{h_{i}})$ is 0. $x_{i,j}$ also has a feature without strength, i.e. $\phi_{M(``\#you")}(x_{i,j},\mathbf{h_{i}})$ = $3$. Function $M(\cdot)$ maps each feature to a unique index.

For predicting the outcome of debate $\mathbf{x_{i}}$, we compute the difference of feature vectors from pro and con in two ways: $\tilde{\Phi}^{p}(\mathbf{x_{i}},\mathbf{h_{i}})=\Phi(\mathbf{x_{i}^{p}},\mathbf{h_{i}})-\Phi(\mathbf{x_{i}^{c}},\mathbf{h_{i}})$ and $\tilde{\Phi}^{c}(\mathbf{x_{i}},\mathbf{h_{i}})=\Phi(\mathbf{x_{i}^{c}},\mathbf{h_{i}})-\Phi(\mathbf{x_{i}^{p}},\mathbf{h_{i}})$. Two decision scores are computed as $f^{p}(\mathbf{x_{i}})=\max_{\mathbf{h_{i}}}[\mathbf{w}\cdot\tilde{\Phi}^{p}(\mathbf{x_{i}},\mathbf{h_{i}})]$ and $f^{c}(\mathbf{x_{i}})=\max_{\mathbf{h_{i}}}[\mathbf{w}\cdot\tilde{\Phi}^{c}(\mathbf{x_{i}},\mathbf{h_{i}})]$. The output is $1$ if $f^{p}(\mathbf{x_{i}})>f^{c}(\mathbf{x_{i}})$ (pro wins); otherwise, the prediction is $-1$ (con wins).

Weights $\mathbf{w}$ are learned during training, while topic strengths $\mathbf{h_{i}}$ are latent variables, and we use Integer Linear Programming to search for $\mathbf{h_{i}}$.

3.2 Learning with Latent Variables

To learn the weight vector $\mathbf{w}$, we use the large margin training objective:

[TABLE]

We consider samples based on difference feature vectors $\tilde{\Phi}^{p}(\mathbf{x_{i}},\mathbf{h_{i}})$ during training, which is represented as $\tilde{\Phi}(\mathbf{x_{i}},\mathbf{h_{i}})$ in Eq. 1 and the rest of this section. $l(\cdot)$ is squared-hinge loss function. $C$ controls the trade-off between the two items.

This objective function is non-convex due to the maximum operation [Yu and Joachims, 2009]. We utilize Alg. 1, which is an iterative optimization algorithm, to search for the solution for $\mathbf{w}$ and $\mathbf{h}$. We first initialize latent topic strength variables as $\mathbf{h_{0}}$ (see next paragraph) and learn the weight vector as $\mathbf{w^{\ast}}$. Adopted from ?), our iterative algorithm consists of two major steps. For each iteration, the algorithm first decides the latent variables for positive examples. In the second step, the solver iteratively searches for latent variable assignments for negative samples and updates the weight vector $\mathbf{w}$ with a cutting plane algorithm until convergence. Global variable $H_{i}$ is maintained for each negative sample to store all the topic strength assignments that give the highest scores during training.666A similar latent variable model is presented in ?) to predict the objection behavior in courtroom dialogues. In their work, a binary latent variable is designed to indicate the latent relevance of each utterance to an objection, and only relevant utterances contribute to the final objection decision. In our case our latent variables model argument strength, and all arguments matter for the debate outcome. This strategy facilitates efficient training while a local optimum is guaranteed.

Topic strength initialization. We investigate three approaches for initializing topic strength variables. The first is based on the usage frequency per topic. If one side uses more arguments of a given topic, then its strength is likely to be strong for them and weak for their opponent. Another option is to initialize all topics as strong for both sides, then $\mathbf{w_{0}}$ learns the association between strong topics and features that lead to winning. The third option is to initialize all topics as strong for winners and weak for losers.

3.3 Features

We group our directly observed linguistic features, roughly ordered by increasing complexity, into categories that characterize various aspects of arguments. For each linguistic feature, we compute two versions: one for the full debate and one for the discussion phase.

Basic features. We consider the frequencies of words, numbers, named entities per type, and each personal pronoun. For instance, usage of personal pronouns may imply communicative goals of the speaker [Brown and Gilman, 1960, Wilson, 1990]. We also count the frequency of each POS tag output by Stanford parser [Klein and Manning, 2003]. Sentiment and emotional language usage is prevalent in discussions on controversial topics [Wang and Cardie, 2014b]. We thus count words of positive and negative sentiment based on MPQA lexicon [Wilson et al., 2005], and words per emotion type according to a lexicon from ?). Moreover, based on the intuition that agreement carries indications on topical alignment [Bender et al., 2011, Wang and Cardie, 2014a], occurrence of agreement phrases (“I/we agree”, “you’re right”) is calculated. Finally, audience feedback, including applause and laughter, is also considered.

Style features. Existing work suggests that formality can reveal speakers’ opinions or intentions [Irvine, 1979]. Here we utilize a formality lexicon collected by ?), which counts the frequencies of formal or informal words in each argument. According to ?), hedges are indicators of weak arguments, so we compile a list of hedge word from ?), and hedging of verbs and non-verbs are counted separately. Lastly, we measure word attributes for their concreteness (perceptible vs. conceptual), valence (or pleasantness), arousal (or intensity of emotion), and dominance (or degree of control) based on the lexicons collected by ?) and ?), following ?), who observe correlations between word attributes and their persuasive effect in online arguments. The average score for each of these features is then computed for each argument.

Semantic features. We encode semantic information via semantic frames [Fillmore, 1976], which represent the context of word meanings. ?) show that arguments of different types tend to employ different semantic frames, e.g., frames of “reason” and “evaluative comparison” are frequently used in making claims. We count the frequency of each frame, as labeled by SEMAFOR [Das et al., 2014].

Discourse features. The usage of discourse connectors has been shown to be effective for detecting argumentative structure in essays [Stab and Gurevych, 2014]. We collect discourse connectives from the Penn Discourse Treebank [Prasad et al., 2007], and count the frequency of phrases for each discourse class. Four classes on level one (temporal, comparison, contingency, and expansion) and sixteen classes on level two are considered. Finally, pleading behavior is encoded as counting phrases of “urge”, “please”, “ask you”, and “encourage you”, which may be used by debaters to appeal to the audience.

Sentence-level features. We first consider the frequency of questioning since rhetorical questions are commonly used for debates and argumentation. To model the sentiment distribution of arguments, sentence-level sentiment is labeled by the Stanford sentiment classifier [Socher et al., 2013] as positive (sentiment score of 4 or 5), negative (score of 1 or 2), and neutral (score of 3). We then count single sentence sentiment as well as transitions between adjacent sentences (e.g. positive $\rightarrow$ negative) for each type. Since readability level may affect how the audience perceives arguments, we compute readability levels based on Flesch reading ease scores, Flesch-Kincaid grade levels, and the Coleman-Liau index for each sentence. We use the maximum, minimum, and average of scores as the final features. The raw number of sentences is also calculated.

Argument-level features. Speakers generally do not just repeat their best argument ad infinitum, which suggests that arguments may lose power with repetition. For each argument, we add an indicator feature (i.e. each argument takes value of $1$) and an additional version with a decay factor of $\exp(-\alpha\cdot t_{k})$, where $t_{k}$ is the number of preceding arguments by a given side which used topic $k$, and $\alpha$ is fixed at $1.0$. Interruption is also measured, when the last argument (of more than 50 words) in a turn is cut off by at most two sentences from opponent or moderator. Word repetition is often used for emphasis in arguments [Cano-Basave and He, 2016], so we measure bigram repetition more than twice in sequential clauses or sentences.

Interaction features. In addition to independent language usage, debate strategies are also shaped by interactions with other debaters. For instance, previous work [Zhang et al., 2016] finds that debate winners frequently pursue talking points brought up by their opponents’. Here we construct different types of features to measure how debaters address opponents’ arguments and shift to their favorable subjects. First, for a given argument, we detect if there is an argument of the same topic from the previous turn by the opponent. If yes, we further measure the number of words of the current argument, the number of common words between the two arguments (after lemmatization is applied), the concatenation of the sentiment labels, and the concatenation of the emotion labels of the two arguments as features; these interactions thus capture interactive strategies regarding quantity speech and sentiment. We also consider if the current argument is of a different topic from the previous argument in the same turn to encode topic shifting behavior.

Feature functions $\phi_{M(\mathtt{feature},\mathtt{strength})}(x_{i,j},\mathbf{h_{i}})$ in § 3.1 only consider the strengths of single arguments. To capture interactions between sides that relate to their relative argument strengths, we add features $\phi_{M(\mathtt{feature},\mathtt{strength^{self},strength^{oppo}})}(x_{i,j},\mathbf{h_{i}})$, so that strengths of pairwise arguments on the same topic from both sides are included. For instance, for topic “execution of innocents”, side pro with strong strength uses an argument of 100 words to address the challenge raised by con with weak strength. We add four grouped features associated with the number of words addressing an opponent: $\phi_{M(``\#words~{}to~{}oppo",``strong,weak")}(x_{i,j},\mathbf{h_{i}})$ is 100, while $\phi_{M(``\#words~{}to~{}oppo",``weak,weak")}(x_{i,j},\mathbf{h_{i}})$, $\phi_{M(``\#words~{}to~{}oppo",``strong,strong")}(x_{i,j},\mathbf{h_{i}})$, and $\phi_{M(``\#words~{}to~{}oppo",``weak,strong")}(x_{i,j},\mathbf{h_{i}})$ are all 0.

3.4 Topic Strength Inference

Topic strength inference is used both for training (Alg. 1) and for prediction. Our goal is to find an assignment $\mathbf{h_{i}^{*}}$ that maximizes the scorer $\mathbf{w^{*}}\cdot\tilde{\Phi}(\mathbf{x_{i}},\mathbf{h_{i}})$ for a given $\mathbf{w^{*}}$. We formulate this problem as an Integer Linear Programming (ILP) instance.777We use LP Solve: http://lpsolve.sourceforge.net/5.5/. Since topic strength assignment only affects feature functions that consider strengths, here we discuss how to transform those functions into the ILP formulation.

For each topic $k$ of a debate $d_{i}$, we create binary variables $r_{k,strong}^{p}$ and $r_{k,weak}^{p}$ for pro, where $r_{k,strong}^{p}=1$ indicates the topic is strong for pro and $r_{k,weak}^{p}=1$ denotes the topic is weak. Similarly, $r_{k,strong}^{c}$ and $r_{k,weak}^{c}$ are created for con.

Given weights associated with different strengths $w_{M(\mathtt{feature},strong)}$ and $w_{M(\mathtt{feature},weak)}$, the contribution of any feature to the objective (i.e. scoring difference between pro and con) transforms from

$w_{M(\mathtt{feature},strong)}\cdot[\sum_{x_{i,j}\in\mathbf{x_{i}^{p}}}\phi_{M(\mathtt{feature},strong)}(x_{i,j},\mathbf{h_{i}})\\ -\sum_{x_{i,j}\in\mathbf{x_{i}^{c}}}\phi_{M(\mathtt{feature},strong)}(x_{i,j},\mathbf{h_{i}})]\\ +w_{M(\mathtt{feature},weak)}\cdot[\sum_{x_{i,j}\in\mathbf{x_{i}^{p}}}\phi_{M(\mathtt{feature},weak)}(x_{i,j},\mathbf{h_{i}})\\ -\sum_{x_{i,j}\in\mathbf{x_{i}^{c}}}\phi_{M(\mathtt{feature},weak)}(x_{i,j},\mathbf{h_{i}})]\\$

to the following form:

$w_{M(\mathtt{feature},strong)}\cdot[\sum_{x_{i,j}\in\mathbf{x_{i}^{p}}}\phi_{M(\mathtt{feature})}(x_{i,j},\mathbf{h_{i}})\cdot r_{k,strong}^{p}\\ -\sum_{x_{i,j}\in\mathbf{x_{i}^{c}}}\phi_{M(\mathtt{feature})}(x_{i,j},\mathbf{h_{i}})\cdot r_{k,strong}^{c}]\\ +w_{M(\mathtt{feature},weak)}\cdot[\sum_{x_{i,j}\in\mathbf{x_{i}^{p}}}\phi_{M(\mathtt{feature})}(x_{i,j},\mathbf{h_{i}})\cdot r_{k,weak}^{p}\\ -\sum_{x_{i,j}\in\mathbf{x_{i}^{c}}}\phi_{M(\mathtt{feature})}(x_{i,j},\mathbf{h_{i}})\cdot r_{k,weak}^{c}]\\$

The above equation can be reorganized into a linear combination of variables $r_{*,*}^{*}$. We further include constraints as discussed below, and solve the maximization problem as an ILP instance.

For features that consider strength for pairwise arguments, i.e. $\phi_{M(\mathtt{feature},\mathtt{strength^{self},strength^{oppo}})}$, we have binary variables $r_{k,strong,strong}^{p,c}$ (strength is strong for both sides), $r_{k,strong,weak}^{p,c}$ (strong for pro, weak for con), $r_{k,weak,strong}^{p,c}$ (weak for pro, strong for con), and $r_{k,weak,weak}^{p,c}$ (weak for both).

Constraints. We consider three types of topic strength constraints for our ILP formulation:

$\bullet$ C1 – Single Topic Consistency: each topic can either be strong or weak for a given side, but not both.

pro: $r_{k,strong}^{p}+r_{k,weak}^{p}=1$; con: $r_{k,strong}^{c}+r_{k,weak}^{c}=1$

$\bullet$ C2 – Pairwise Topic Consistency: for pairwise arguments from pro and con on the same topic, their joint assignment is true only when each of the individual assignments is true. C2 applies only for features of pairwise arguments.

$r_{k,strong,strong}^{p,c}=r_{k,strong}^{p}\wedge r_{k,strong}^{c}$;

$r_{k,strong,weak}^{p,c}=r_{k,strong}^{p}\wedge r_{k,weak}^{c}$;

$r_{k,weak,strong}^{p,c}=r_{k,weak}^{p}\wedge r_{k,strong}^{c}$;

$r_{k,weak,weak}^{p,c}=r_{k,weak}^{p}\wedge r_{k,weak}^{c}$

$\bullet$ C3 – Exclusive Strength: a topic cannot be strong for both sides. This constraint is optional and will be tested in experiments. $r_{k,strong}^{p}+r_{k,strong}^{c}\leq 1$

3.5 Argument Identification

In order to identify the topics associated with a debate and the contiguous chunks of same-topic text that we take to be arguments, for each separate debate we utilize a Hidden Topic Markov Model (HTMM) [Gruber et al., 2007] which jointly models the topics and topic transitions between sentences. For details on HTMM, we refer the readers to ?).

The HTMM assigns topics on the sentence level, assuming each sentence is generated by a topic draw followed by word draws from that topic, with a transition probability determining whether each subsequent sentence has the same topic as the preceding, or is a fresh draw from the topic distribution. Unlike the standard HTMM process, however, we presume that while both sides of a debate share the same topics, they may have different topic distributions reflecting the different strengths of those topics for either side. We thus extend the HTMM by allowing different topics distributions for the pro and con speech transcripts, but enforce shared word distributions for those topics. To implement this, we first run HTMM on the entire debate, and then rerun it on the pro and con sides while fixing the topic-word distributions. Consecutive sentences by the same side with the same topic are treated as a single argument.

4 Experimental Results

4.1 Experimental Setup

We test via leave-one-out for all experiments. For logistic regression classifiers, L2 regularization with a trade-off parameter of 1.0 is used. For Support Vector Machines (SVM) classifiers and our models, we fix the trade-off parameter between training error and margin as 0.01. Real-valued features are normalized to $[0,1]$ via linear transformation.

Our modified HTMM is run on each debate for between 10 and 20 topics. Topic coherence, measured via ?), is used to select the topic number that yields highest score. On average, there are 13.7 unique topics and about 322.0 arguments per debate.

4.2 Baselines and Comparisons

We consider two baselines trained with logistic regression and SVMs classifiers: (1) ngrams, including unigrams and bigrams, are used as features, and (2) audience feedback (applause and laughter) are used as features, following ?). We also experiment with SVMs trained with different sets of features, presented in § 3.3.

4.3 Results

The debate outcome prediction results are shown in Table 1. For our models, we only display results with latent strength initialization based on frequency per topic, which achieves the best performance. Results for different initialization methods are exhibited and discussed later in this section. As can be seen, our model that leverages learned latent topic strengths and their interactions with linguistic features significantly outperform the non-trivial baselines888For baselines with logistic regression classifiers, the accuracy is 63.6 with ngram features, and 58.5 with audience feedback features. (bootstrap resampling test, $p<0.05$). Our latent variable models also obtain better accuracies than SVMs trained on the same linguistic feature sets. Without the audience feedback features, our model yields an accuracy of 72.0%, while SVM produces 65.3%. This is because our model can predict topic strength out of sample by learning the interaction between observed linguistic features and unobserved latent strengths. During test time, it infers the latent strengths of entirely new topics based on observable linguistic features, and thereby predict debate outcomes more accurately than using the directly observable features alone. Using the data in ?) (a subset of ours), our best model obtains an accuracy of 73% compared to 65% based on leave-one-out setup.

As mentioned above, we experimented with a variety of latent topic strength initializations: argument frequency per topic (Freq); all topics strong for both sides (AllStrong); strong just for winners (AllStrongwin); and Random initialization. From Table 2, we can see that there is no significant difference among different initialization methods. Furthermore, the strength constraints make little difference, though their effects slightly vary with different initializations. Most importantly, C3 (the constraint that topics cannot be strong for both sides) does not systematically help, suggesting that in many cases topics may indeed be strong for both sides, as discussed below.

5 Discussion

In this section, we first analyze argument strengths for winning and losing sides, followed by a comparison of these results with human evaluations (§ 5.2). We then examine the interactive topic shifting behavior of debaters (§ 5.3) and analyze the linguistic features predictive of debate outcome, particularly the ones that interact with topic strength (§ 5.4). The results are reported by training our model on the full dataset. Initialization of topic strength is based on usage frequency unless otherwise specified.

5.1 Topic and Argument Usage Analysis

We start with a basic question: do winning sides more frequently use strong arguments? For each side, we calculate the proportion of strong and weak topics as well as the total number of strong and weak arguments on each side. Figure 2 shows that under all three topic strength initializations, our model infers a greater number of strong topics for winners than for losers. This result is echoed by human judgment of topic strength, as described in § 5.2. Similarly, winners also use significantly more individually strong arguments.

As can be seen in Table 2, the C3 constraint, that a topic be strong for at most one side, only increased accuracy for one initialization case. This indicates that, in general, the model was improved by allowing some topics to be strong for both sides. Interestingly, while the majority (53%) of topics are strong for one side and weak for the other, about a third (31%) of topics are inferred as strong for both sides. While it is clear what it means for a topic to be strong for one side and not the other (as in our death penalty example), or weak for both sides (as in a digression off of the general debate topic), the importance of both-strong for prediction is a somewhat surprising result. Figure 3 illustrates an example as judged by our model. What this shows is that even on a given topic within a debate (Syrian refugees: resettlement), there are different subtopics that may be selectively deployed (resettlement success; resettlement cost) that make the general topic strong for both sides in different ways. For subsequent work, a hierarchical model with nested strength relationships [McCombs, 2005, Nguyen et al., 2015] can be designed to better characterize the topics.

Lastly, we display the usage of strong arguments during the course of debates in Figure 4. Each debate is divided into opening statements, two interacting phases (equal number of turns), and closing statements. Similar usage of strong arguments are observed as debates progress, though a slight, statistically non-significant drop is noted in the closing statement. One possible interpretation is that debaters have fully delivered their strong arguments during opening and interactions, while only weaker arguments remain when closing the debates.

5.2 Human Validation of Topic Strength

Here we evaluate whether our inferred topic strength matches human judgment. We randomly selected 20 debates with a total of 268 topics. For each debate, we first displayed its motion and a brief description constructed by IQ2. Then for each topic, the top 30 topic words from the HTMM model were listed, followed by arguments from pro and con. Note that debate results were not shown to the annotators.

We hired three human annotators who are native speakers of English. Each of them was asked to first evaluate topic coherence by reading the word list and rate it on a 1-3 scale (1 as incoherent, 3 as very coherent). If the judgment was coherent (i.e. a 2 or 3), they then read the arguments and judged whether (a) both sides are strong on the topic, (b) both sides are weak, (c) pro is strong and con is weak, or (d) con is strong and pro is weak.

54.9% of the topics were labeled as coherent by at least two annotators, although since topics are estimated separately for each debate, even the less coherent topics generally had readily interpretable meanings in the context of a given debate. Among coherent topics, inter-annotator agreement for topic strength annotation had a Krippendorff’s $\alpha$ of 0.32. Judging argument strength is clearly a more difficult and subjective task.

Nevertheless, without knowing debate outcomes, all three human judges identified more strong topics for winning sides than losing sides. Specially, among the coherent topics, a (macro-)average of 44.4% of topics were labeled as strong for winners, compared to 30.1% for losers. This echoes the results from our models as illustrated in Figure 2.

Furthermore, we calculate the correlation between topic strength inferred by our models and the ones labeled by each human judge using Cohen’s $\kappa$. The results are illustrated in Figure 5, which shows our three different initializations, with and without learning. The highest human $\kappa$ is also displayed. Our trained models clearly match human judgments better than untrained ones.

5.3 Topic Shifting Behavior Analysis

Within competitive debates, strategy can be quite interactive: one often seeks not just to make the best arguments, but to better the opponent from round to round. Agenda setting, or shifting the topic of debate, is thus a crucial strategy. An important question is therefore: do debaters strategically change topics to ones that benefit themselves and weaken their opponent? According to the HTMM results, debaters make 1.5 topical shifts per turn on average. Both winning and losing teams are more likely to change subjects to their strong topics: winners in particular are much more likely to change the topic to something strong for them (61.4% of shifts), although debate losers also attempt this strategy (53.6% of shifts).

A more sophisticated strategy is if the debaters also attempt to put their opponents at a disadvantage with topic shifts. We consider the topic strengths of a current argument for both the speaker (“self”) and their “opponent”, as well as the strength of the following argument. The top 3 types of shifts are listed in Table 3. As can be seen, winners are more likely to be in a strong (for them) and weak (for the opponent) situation and to stay there, while losers are more likely to be in the reverse. Both sides generally stay in the same strength configuration from argument 1 to argument 2, but winners are also likely (row 3) to employ the strategy of shifting from a topic that is strong for both sides, to one that is strong for them and weak for the opponent.

5.4 Feature Analysis

Lastly, we investigate the linguistic features associated with topics of different strengths that affect the audience. Table 4 displays some of the 50 highest weighted features that interact with strong and weak topics. Personal pronoun usage has been found to be related to communicative goals in many previous studies [Brown and Gilman, 1960, Wilson, 1990]. We find that strong topics are associated with more first person plurals, potentially an indicator of group responsibility [Wilson, 1990]. On the other hand, our model finds that weak topics are associated with second person pronouns, which may be arguments either attacking other discussants or addressing the audience [Simons and Jones, 2011]. For sentiment, previous work [Tan et al., 2016] has found that persuasive arguments are more negative in online discussions. Our model associates negative sentiment and anger words with strong topics, and neutral and joyful languages with weak topics.

In terms of style and discourse, debaters tend to use more formal and more concrete words for arguments with strong topics. By contrast, arguments with weak topics show more frequent usage of words with intense emotion (higher arousal scores), and contrast discourse connectives. Figure 6 shows how some of these features differ between winners and losers, illustrating the effects on outcome via strong or weak arguments in particular.

Interaction features also play an important role for affecting audiences’ opinions. In particular, debaters spend more time (i.e. use more words) addressing their opponents’ arguments if it is a strong topic for their opponents. But even for weak topics, it appears helpful to address opponents’ arguments.

6 Related Work

Previous work on debate and persuasion has studied the dynamic of audience response to debates and the rhetorical frames the speakers use [Boydstun et al., 2014]. However, this work is limited by the scarcity of data and does not focus on the interactions between content and language usage. Topic control, operationalized as the tendency of one side to adopt or avoid the words preferentially used by the other, is investigated in ?) to predict debate outcome using the Intelligence Squared data. Our work complements theirs in examining topic interactions, but brings an additional focus on the latent persuasive strength of topics, as well as strength interactions. ?) examine various structural and linguistic features associated with persuasion on Reddit; they find that some topics correlates more with malleable opinions. Here we develop a more general model of latent topic strength and the linguistic features associated with strength.

Additional work has focused on the influence of agenda setting — controlling which topics are discussed [Nguyen et al., 2014], and framing [Card et al., 2015, Tsur et al., 2015] — emphasizing certain aspects or interpretations of an issue. ?) study the syntactic aspects of framing, where syntactic choices are found to correlate with the sentiment perceived by readers. Based on the topic shifting model of ?), ?) find that changing topics in presidential primary debates positively correlates with the candidates’ power, which is measured based on their relative standing in recent public polls. This supports our finding that both sides seek to shift topics, but that winners are more likely to shift and shift to topics which are strong for them but weak for their opponents.

Our work is in line with argumentation mining. Existing work in this area focuses on argument extraction [Moens et al., 2007, Palau and Moens, 2009, Mochales and Moens, 2011] and argument scheme classification [Biran and Rambow, 2011, Feng and Hirst, 2011, Rooney et al., 2012, Stab and Gurevych, 2014]. Though stance prediction has also been studied [Thomas et al., 2006, Hasan and Ng, 2014], we are not aware of any work that extracts arguments according to topics and position. Argument strength prediction is also studied largely in the domain of student essays [Higgins et al., 2004, Stab and Gurevych, 2014, Persing and Ng, 2015]. Notably, none of these distinguishes an argument’s strength from its linguistic surface features. This is a gap we aim to fill.

7 Conclusion

We present a debate prediction model that learns latent persuasive strengths of topics, linguistic style of arguments, and the interactions between the two. Experiments on debate outcome prediction indicate that our model outperforms comparisons using audience responses or linguistic features alone. Our model also shows that winners use stronger arguments and strategically shift topics to stronger ground. We also find that strong and weak arguments differ in their language usage in ways relevant to various behavioral theories of persuasion.

Acknowledgments

This work was supported in part by National Science Foundation Grant IIS-1566382 and a GPU gift from Nvidia. We thank the TACL reviewers for valuable suggestions on various aspects of this work.