The evolution of argumentation mining: From models to social media and emerging tools

TL;DR

This paper reviews the evolution of argumentation mining, highlighting the shift from traditional models to social media applications, and proposes a flexible framework for extracting structured arguments from noisy, unstructured social media text.

Contribution

It bridges theoretical argumentation models with pragmatic social media schemes and introduces a conceptual architecture for adaptable argumentation mining.

Findings

Existing approaches vary in effectiveness for social media data

Combining multiple tasks improves argumentation mining performance

Proposed framework enhances flexibility and extensibility in social media contexts

Abstract

Argumentation mining is a rising subject in the computational linguistics domain focusing on extracting structured arguments from natural text, often from unstructured or noisy text. The initial approaches on modeling arguments was aiming to identify a flawless argument on specific fields (Law, Scientific Papers) serving specific needs (completeness, effectiveness). With the emerge of Web 2.0 and the explosion in the use of social media both the diffusion of the data and the argument structure have changed. In this survey article, we bridge the gap between theoretical approaches of argumentation mining and pragmatic schemes that satisfy the needs of social media generated data, recognizing the need for adapting more flexible and expandable schemes, capable to adjust to the argumentation conditions that exist in social media. We review, compare, and classify existing approaches, techniques and tools, identifying the positive outcome of combining tasks and features, and eventually propose a conceptual architecture framework. The proposed theoretical framework is an argumentation mining scheme able to identify the distinct sub-tasks and capture the needs of social media text, revealing the need for adopting more flexible and extensible frameworks.

Tables

Table 1. Table 1: A synopsis of logical schemes and computational theories based on the degree they can meet the needs in a modern NLP environment. Link relation - if the connection distinctive components are explicitly evaluated, Complexity level - is assessed based on the number of components each theory includes.

Author(s)	Link Relation	Complexity Level	Application in noisy data	Application in AM
Whately	No	High	Low	Medium
Beardsley	No	Low	Medium	High
Toulmin	Yes	High	Medium-Low	High
Freeman	Yes	Medium	Medium	Medium
RST	Yes	Open	High	High
Pollock	Yes	High	Low	Medium
Dung	Yes	High	Low	Medium
Krause	Yes	High	Low	Medium
Parsons	Yes	High	Low	Medium
New Rhetoric	Yes	High	Low	Medium

Table 2. Table 2: Ck - Cohen’s kappa [ 75 ] , Ka - Krippendorff’s alpha [ 76 ] , Dice [ 77 ] , Fk - Fleiss kappa [ 78 ] . In [ 67 ] , the IAA is the average of the three sub-tasks. Details of datasets that have been used in recent research papers, displaying the distinctive tasks in AM, the different sizes of the datsaets, the plethora of topics and the range in IAA metrics.

Authors	Task(s)	Size	Topic	#Annot.	IAA	#Classes
Addawood and Bashir [3]	Argument Detection	3000 tweets	Apple/FBI encryption debate	2	0.67 Ck	2
	Evidence Categorization				0.79 Ck	6
Bosc et al. [61]	Argument Detection	4000 tweets	5 debate topics	3	0.81 Ka	2
	Relations Identification				0.67 Ka	3
Deturck et al. [62]	Argument Detection	70 M tweets	Festival evaluation	-	-	-
Dusmanu et al. [63]	Argument Detection	1887	Grexit / Brexit	2	0.77 Ck	2
	Factual vs opinion				0.73 Ck	2
	Source Identification				0.84 Dice	2
Sendi and Latiri [64]	Argument Detection	70 M tweets	Festival evaluation	-	-	-
Dufour et al. [65]	Argument Detection	70 M tweets	Festival evaluation	-	-	-
Cocarascu and Toni [66]	Relations Identification	30	Hillary Clinton’s FBI investigation	3	0.11 Fk	3
		840		NG	NG	NG
Ma et al. [67]	Relations Identifications	2520	30 debate topics	3	0.81 MV	2
Mohammad et al. [68]	Stance Detection	4163	5 debate topics	5-8	0.82 MV	3
Zarella and Marsh [69]	Stance Detection	4163	5 debate topics	5-8	0.82 MV	3
Wei et al. [70]	Stance Detection	4163	5 debate topics	5-8	0.82 MV	3
Ebrahimi et al. [71]	Stance Detection	78000+	Donald Trump	-	-	-
Lai et al. [72]	Stance Detection	963 triplets	Italian Constitution refenderum	2-5		3
Johnson and Goldwasser [73]	Stance Detection	99161	16 debate topics	-	-	-
Addawood et al. [60]	Stance Detection	3000	Apple/FBI encryption debate	2	0.64 Ck	3
	Topic Classification				0.70 Ck	4
Konstantinovskiy et al. [74]	Claim Detection	4080	Various political issues	5	0.46 Ka	7
					0.7 Ka	2

Table 3. Table 3: Features used for AM tasks in social media. IR = Information Retrieval applied as lexicon-based queries, PoS = Part of Speech, TM = Topic Modeling applied with the technique of Latent Dirichlet Allocation (LDA), LIWC = Linguistic Enquiry and Word Count [ 80 , 81 ]

	Basic (Lexical)	Semantic	Sentiment	Subjectivity	Twitter	Other
Addawood and Bashir [3]	n-gram, length, question, exclamation	PoS, LIWC summary, variables	LIWC, sentiment, lexicon	Clue lexicon	followers, friends, user activity, URL+title, hashtags, verified account, mentions	Psycometric, LIWC
Bosc et al. [61]	n-gram, punctuation, tokens, capitalization	PoS			smileys
Deturck et al. [62]	tokens, lemmas	PoS	TextBlob features	TextBlob features		word embeddings
Dusmanu et al. [63]	n-gram	Syntactic tree, parse trees, dependency relations WordNet synset	AlchemyAPI		Punctuations, emoticons
Sendi and Latiri [64]	IR, TM		NRC emotion lexicon
Dufour et al. [65]	punctuation,	personal pronouns	Emotion words		Emoticons, hashtags
Cocarascu and Toni [66]	n-grams					word embeddeings argumentativeness
Ma et al. [67]	n-gram, Relevance feature, Okapi BM25	Topic-Independent Claim-Related Lexicon, Topic-Dependent Claim-Oriented Lexicon		Controversy lexicon	URLs, retweet reply	TwitStan WikiClaim TwitArgument
Mohammad et al. [68]	n-grams	PoS	1)NRC Emotion Lexicon, 2)Hu and Liu Lexicon	MPQA Subjectivity Lexicon	hashtag	Target of interest, encodings
Zarella and Marsh [69]						word embeddings
Wei et al. [70]						word embeddings, target content
Ebrahimi et al. [71]	n-grams, linguistic patterns		1) LIWC2007 2) VADER
Lai et al. [72]					social media network communities, Hashtags, Mentions, Replies
Johnson and Goldwasser [73]	Words frequency	keyword-based heuristic	OpinionFinder 2.0		Temporal activity
Addawood et al. [60]	n-grams	PoS	Subjectivity lexicon	MPQA Subjectivity Lexicon	RT, title, mention, verified account, url, followers, following, posts, hashtag	Argumentativeness, source type
Konstantinovskiy et al. [74]		PoS, NER, tf-idf				sentence embeddings

Table 4. Table 4: The results of the supervised ML algorithms for the task of argument detecton. Un-supervised methods are not included. RF = Round Forest, LR = Logistic Regression, DT = Decision Tree, SVM = Support Vector Machine, NB = Naive Bayes. Rounding took place in order the results to be displayed in the same scale.

Authors	Algorithm	Features	Precision	Recall	F1
Addawood and Bashir [3]	DT	n-gram	0.72	0.69	0.66
	SVM	n-gram	0.81	0.78	0.77
	NB	n-gram	0.70	0.67	0.64
	DT	all features	0.87	0.87	0.87
	SVM	all features	0.89	0.89	0.89
	NB	all features	0.79	0.79	0.85
Bosc et al. [61]	LR	lexical	-	-	0.64
	LR	lexical + semantic	-	-	0.66
	LR	all features	-	-	0.67
Dusmanu et al. [63]	RF	n-gram	0.76	0.69	0.71
	LR	n-gram	0.76	0.71	0.73
	LR	all features	0.80	0.77	0.78

Table 5. Table 5: The different sub-tasks that have been accomplished in the context of relations identification task. MAP = Mean Average Precision, P@5 = Precision@5, P@10 = Precision@10. Rounding took place in order the results to be displayed in the same scale.

Author	Scope	Task	Algorithms	Metric	Score
Bosc et al. [61]	macro-analysis	support / attack	EOP + RTE	F1 support	0.17
				F1 attack	0.0
			LSTM	F1 support	0.20
				F1 attack	0.16
Cocarascu and Toni [66]	macro-analysis	support / attack / neither	LSTM		dataset [83]	dataset [82]
				P	0.59	0.97
				R	0.97	0.90
				F1	0.73	0.94
Ma et al. [67]	micro & macro analysis	topic relatedness, support/attack, arguable reason	SVM light	MAP	0.59	0.50
				P@5	0.53	0.51
				P@10	0.48	0.44

Table 6. Table 6: The results of the supervised and weakly-supervised ML approaches that have been followed for the stance detection in social media text. Fully supervision on [ 69 ] , [ 68 ] , [ 60 ] . Weakly supervision on [ 70 ] , [ 71 ] , [ 73 ] , [ 72 ] . The [ 69 ] , [ 68 ] , [ 98 ] , [ 71 ] and [ 60 ] are applied on the same dataset. RNN = Recurrent Neural Network

Author	# classes	Algorithm	Metric	Score
Zarella and Marsh [69]	Favor/against/neither	RNN	F1	0.68
Mohammad et al. [68]	Favor/against/neither	linear-kernel SVM	F-micro	0.70
			F-macro	0.59
Wei et al. [98]	Favor/against/neither	Neural network	F1	0.56
Wei et al. [70]	Favor/against	Neural network	F1	0.71
Ebrahimi et al. [71]	Favor/against/neither	Linear-kernel SVM	F macro	0.57
Johnson and Goldwasser [73]	Favor/against	Probabilistic Soft Logic	A	0.86
Lai et al. [72]	Favor/against	SVM	F-macro	0.90
Addawood et al. [60]	Favor/against/neutral	SVM	P	0.90
			R	0.90
			F1	0.90

Table 7. Table 7: The results of the supervised ML approaches that have been followed for reliability-related tasks in AM, in text derived from social media. For the task of source identification a rule-based approach is followed. str match = string matching. h = heuristic algorithm

Author	Task	# classes	Algorithm	Metric	Score
Addawood et al. [3]	Evidence Classification	6	SVM	F1-macro	0.83
Dusmanu et al. [63]	Factual vs opinion	2	LR	P	0.81
				R	0.79
				F1	0.80
	Source Identification	NA	str match + h.	P	0.69
				R	0.64
				F1	0.67
Konstantinovskiy et al. [74]	Claim detection	2	LR	P	0.88
				R	0.80
				F1	0.83
		7	LR	P-micro	0.71
				R-micro	0.73
				F1-micro	0.70
				P-macro	0.61
				R-macro	0.44
				F1-macro	0.48

Table 8. Table 8: A summarization of the existing NLP tools that can enhance the process of AM. The table includes tools in the wider NLP area which can be integrated at any stage of an AM pipeline.

Tool	Web UI	Manual Annotation	Arg Retrieval	Arg Evaluation
WebAnno [112]	Yes	Yes
BRAT [102]	Yes	Yes
GraPAT [105]	Yes	Yes
DiGAT [109]	Yes	Yes
MARGOT [113]	Yes		Yes	Yes
OVA+ [114]	Yes	Yes
TOAST [115]	Yes		Yes	Yes
GATE Temware [108]	Yes	Yes
Args [116]	Yes		Yes	Yes
ArgumenText [117]	Yes		Yes	Yes
Rationale [118]	Yes	Yes

Equations

$$(a_{\text{ij}},t_i,c_j)$$

$$(h_i,s_{\text{ijkl}},d_j,r_k,t_l)$$

Full text

††journal: Information Processing & Management

1 Introduction

Argumentation mining (AM) is identified as a multidisciplinary research topic, with roots on rhetoric and philosophy [1], which gained the interest of the scientific community because of its potential when novel Artificial Intelligence (AI) algorithms and techniques are exploited. The recent advances in Machine Learning (ML) in combination with the emergence of social web can enable impressive progress in different scientific fields with great impact on commercial applications. An AM system has the capacity to mine and analyse a great volume of text data through a variety of sources, providing tools for policy-making and socio-political sciences [2, 3, 4], software engineering [5], while it opens new horizons for the broader area of business, economics and finance, with the digital marketing being the most promising field [6, 7, 8, 9].

Opinion mining and sentiment analysis could be characterized as the predecessors of AM in a simplified form and their limits have already been questioned in the effort for seeking a deeper understanding of the human reasoning [10, 11]. As AM we define a series of actions that could be independent or connected to each other and they are relevant to the tasks of detection, extraction and evaluation of arguments, where argument is a piece of text offering evidence or reasoning in favor or against a specific topic.

The core of the human reasoning is the argumentation process and AM-related tasks attempt to solve a series of problems such as the detection of an argumentative stance towards a specific object, the analysis and evaluation of the argument’s components and the detection of possible relations between them. The cohesion in the components of the arguments and the existence of backing in the claims is a major objective in argument mining because these characteristics have the ability to alter a claim to an argument. For human beings the interpretation of an argument is realized instantly without needing any special effort because of their skill to grasp the context of the information and to accomplish connections with previous experiences (facts, opinions, feelings).

This ability of combining multiple sources of information is missing in existing argument models as it is difficult even for human annotators to agree upon specific guidelines for the modeling of an argument. Furthermore, identifying the thin line between implicit and explicit arguments is harsh as often annotators are implicitly driven by the context. Probably that is also the reason why the majority of research is focused on structured data like law text [12, 13, 14], scientific text [15, 16, 17], formal debates [18] or news articles [19, 20] instead of unstructured text like informal discourse and web-user generated data, although recently there have been some notable research endeavors towards this direction, which are presented the section 3.

Annotating and automatically analysing arguments from Web with great heterogeneity of contents and diversity of jargon is a challenging task. Arguments in social media and informal discourse are sometimes implicit, meaning that the logical structure of an argument’s components (premises, claims, warrants) are not always spelled out and instantly distinguishable, hence, analysis must take place in order to determine the distinctive components. In text derived from social media arguments frequently are missing, as it is common a tweet or a facebook post to contain just a stance on a specific topic without supporting it with evidence or reasoning.

The decoding of the human reasoning process into computer language is a challenging task as it consists of many subprocesses that are difficult to be separated and analysed. The medium for arguing for human beings is natural language, whereas the input for ML algorithms and techniques should be distinct, structured and composed with well-established rules. A wide range of methodologies have been applied for modeling natural language, such as explicit distinctive components [1], argumentative zoning [21], tree structures, dialog-oriented diagrams [22], serial structure of arguments [23, 24] and modifications to simpler structures of existing schemes [25, 26].

However, the claim or other parts of an argument might be implicit [27, 8, 28, 29, 9] and tacit assumptions or premises (enthymemes) take place related to commonsense reasoning. This process is named completion or enthymematic argumentation and takes place often and unconsciously in casual discourse that can be found in Web-generated data. The distinction between explicit and implicit argument lies on the presence of certain syntactic constructions or lexemes (such as conjunctions). Implicit arguments, where lack of these characteristics is noticed, can be identified through previously gained knowledge and logical inference.

This a-priori knowledge is extremely difficult to be expressed through conventional argumentation schemes, which demand a strict structure of the components of the argument. The early approaches [1, 30] were focusing on the philosophical aspect of the argument, whereas modern approaches consider unstructured data and implicit relations between the components of the argument [27, 8, 28, 29, 9]. For example, in Toulmin’s model [1, 31] (figure 3), which has a great impact on modern argumentation schemes, a detailed microstructure is proposed with six specified components, 1) an indisputable datum, 2) a subjective claim on the foundation of datum, 3) a warrant that links them imposed by logical inference, 4) the backing of the justification 5) leading to a degree of confidence (qualifiers) as long as 6) a rebuttal cannot withstand the claim. Data that do not have a structured, well-specified format are hard to be represented by such a strict model.

Breaking down arguments deriving from web or from casual discourse is a demanding task with doubts if it is even feasible. An evidence of the previous statement is the fact that the field of opinion mining thrives in social media data [32, 33] and especially in twitter [34, 35], on the contrary only limited research has been conducted on AM in unstructured data and fewer frameworks have been designed able to capture the special features of social media. In 5.1 we propose a conceptual framework able to capture the specific features of social media text and also enhance other tasks in the wider area of NLP with the use of argumentative features.

There is a small number of related works surveying argumentation mining. Peldszus and Stede [36] realized a thorough revision of models and diagrams for arguments, able to be exploited in the context of AM. We provide the necessary background for argumentation schemes and we evaluate them based on their suitability on social media text, but we do not devote the entire paper to this topic. Both [37] and [38], surveyed a big spectrum of the AM field, describing models, corpora and methods, but they overlook the special nature of social media and the special features they present. On contrary, we focus on text derived from social media and our entire approach is based on the characteristics of the social media; chaotic nature, noisy text, vague claims, complicated network relations, implicit premises, etc. While the authors in [39] have addressed AM in web-generated data, they provided limited information about the connection of AM with its distinctive sub-tasks or other NLP tasks. Instead, we perceive AM as an entire pipeline with distinctive sub-tasks, able to be both stand-alone and correlate with each other, and therefore boost existing tasks such as sentiment analysis, topic modeling, rumour identification, etc.

Our contribution lies on the following main axes: bridge the gap between areas that are interrelated, but without having explicit connection, illustrate the current methods, tasks and tools that exist in the wider area of AM, and stress the importance of automatic AM mechanisms and tools. This extensive survey paper illustrates the evolution of argumentation and reveals the formation of AM as a scientific field through a complete review of its roots and the needs that currently form it, followed by a thorough presentation and comparison of the existing AM approaches in text derived from social media. The existing tools for the wider area of NLP are presented, assessing their potential use in AM, followed by the presentation of the tools that have been specifically built for argumentation. Furthermore, a complete conceptual architecture for AM is proposed, covering its different subtasks and possible relations with other fields.

In the next section, the most influential logical schemes are presented as well as the first attempts of connecting the argumentation process with AI. Every aspect of the AM problem is presented in section 3 with a focus on noisy data derived from social media. Section 4 presents the existing tools in the field of AM and classifies them into two categories: general-purpose NLP tools and tools specifically designed for the task of AM. In the last section a new conceptual AM framework for handling text derived from social media is proposed and eventually, future direction for the prosperous field of AM are provided.

2 Argumentation Theory and Models

Although argumentation mining as a term was first introduced in 2009 by Palau and Moens [40], the act of argumentation and its effects are studied since the 4th century BC [41]. Since then, many approaches on studying argumentation have been studied and many theories, schemes, and diagrams have been developed.

The primary factor that has led to the creation of novel evaluation and visualization techniques for argument representation is the need for simple, but effective ways to break down, analyse and eventually better understand arguments. Argumentation can reach a high level of complexity, thus simpler forms of representation are needed. The process of argument illustration and fragmentation is a fundamental concept in AM, where the arguments are inspected, evaluated and eventually expressed in a binary format, capable to be interpreted from different algorithms. In the next subsection, significant argumentation theories are presented, followed by the first research works that realize the connection of argumentation with AI.

2.1 Logical Schemes / Diagrams

Argumentation holds its roots in rhetoric and philosophy thus argumentation diagrams have been developed as aiding tools for the analysis of arguments in well-structured documents and apart from its original role of teaching the reasoning process without falling into logical fallacies, Reed et al. [21] have also stressed their capabilities as analytical tools for meta-philosophy purposes. Diagram techniques started as practical tool for teaching logic and then developed in a more refined method that is used as a concept idea for the modeling of an argument. Logical diagrams have boosted the field of informal logic, as they offer a tool for analysing and evaluating arguments used in everyday life in a much more pragmatic environment comparing to formal logic. A detailed review on argumentation diagrams and their connection with other fields such as formal and informal logic, law, and artificial intelligence is presented in [21]. As the goal of this review paper is neither the connection of argument diagrams to modern AM techniques as in [36], nor the introduction of a classification system for argumentation schemes [42] in this subsection a synopsis of the five more influential diagrams is presented and they are evaluated based on their suitability for the tasks of AM in noisy text. It has to be stressed that the AM diagrams have not been specifically designed to serve the modern construction of arguments as expressed in social media, as the tasks of detection, classification and evaluation of argumentative content in noisy text require more flexible schemes, such the one proposed in the subsection 5.1.

One of the first uses of diagrams in argumentation took place in 1857 by Whately [43], opposing to the enumeration of technical rules, in an effort of simplifying the teaching method of argumentation in his era. Whately’s diagram theory is based on the concept of identifying the concluding assertion and tracing the reasoning backwards, grounding the original assertion and eventually forming a tree with assertions and proofs. In Figure 1, the conclusion is represented as the root of the tree, and the assertions are located underneath. In the classical example of Socrates syllogism (Socrates is a man, all men are mortal, therefore Socrates is mortal), the conclusion (Socrates is mortal) would be the root of the tree and the two premises (p1- Socrates is a man, p2 – all men are mortal) would be the two leaves of the tree. The complexity of the reasoning process and the depth of the tree are proportionate, and could lead to complicated process that requires both well-structured arguments and experienced annotators.

Another significant work in the field of Argumentation is conducted by Beardsley in [44] introducing a designing principal that is applied until today; representation of the argument’s distinctive statements as linked nodes. As the nodes can be connected with each other with different ways, he created three basic classes 1) convergent, 2) divergent and 3) serial arguments. In Figure 2, a logic flow is illustrated depicting serial linking between premises, leading to a convergence for the final argument. In the example of the convergent argument, different premises eventually contribute towards the establishment of a reliable and robust argument, supported with enough backing.

The only flaw in Beardsley’s theory is the lack of support between the statements in the nodes. Thus, the statements which form the argument are considered flawless and there are not subjects of support, debate or evaluation, hence it cannot be applied in ambiguous, implicit or imperfect arguments.

In 1958, Toulmin [45] suggested one very influential scheme until today, examining the role that different utterances might have in the persuasive perspective of the argument. In Toulmin’s model, six functional roles were suggested, namely datum, claim, warrant, backing, qualifiers, and rebuttal.

In Figure 3 an example of legal nature is expressed through the Toulmin’s model. The warrant in the depicted example (“A manborn in Bermuda will generally be a British subject”) supports adequately the initial datum (“Harry was born in Bermuda”) as it includes a real strong backing using a legal framework (“The following statues and other legal provisions”). The above distinctive components are enhanced through the defy of a possible counter-argument (“Both his parents were aliens..”) and eventually lead with certainty expressed through the qualifier (So, presumably) to the conclusion (“Harry is a British subject”).

The novelty on his approach lies on the fact that it requires the assignment of a predefined characterization for the cognitive connection between the different components of the argument. Through his proposed scheme, Toulmin managed to handle enthymematic relations by defining different aspects of a syllogism and connecting the inference with the warrant. The proposed scheme it is widely used in AM and a modification has even been applied in web-generated data [39], however the small IAA in some topics indicate the difficulty of applying a such complex model in heterogeneous text.

Another argumentation theory that still has a strong influence until today as recent research works [46, 47] adopt its scheme in a data driven approach for AM tasks, is developed by Freeman [48]. Freeman’s theory could be characterized as an upgrade of the Beardley’s theory, as it uses the scheme of inductive/deductive reasoning and enhances it with the concept of modality, which indicates the strength of induced conclusion by the premises. It could be said that the term of modality is an adjustment of the Toulmin’s qualifier, but with a focus on the evaluation of the argument. Both the concepts of reasoning flow and argument strength have the potential to enhance the AM tasks as they offer new unprecedented tasks that have not been tested, as only sentiment flows in AM have been researched [49] until now.

The introduction of a prominent text-organization theory took place in the 1980’s by Mann [50], aiming at the organization of the text into different regions. Each region has a central part (nucleus) that is essential for the comprehension of the text, and a number of satellites containing additional information about the nucleus. The nucleus and the satellite are correlated with each other through different relations (circumstance, elaboration, evidence, etc.) which can be changed, manipulated, added or subdivided depending on the topic and the task at hand. The nucleus-satellite distinction is applied recursively until every entity of the discourse is a part of a RST relation and eventually a tree-structure hierarchy is created as depicted in Figure 4, where the theorem of the perception of apparent motion (initial nucleus) is supported by a set premises, where each premise sequentially is expressed through the model of nucleus-satellite expressing a specific relation (preparation, condition, means). The application of RST has improved the performance of sentiment polarity classification when enhanced with argumentation [51], but the authors state the need for further in-depth evaluation for its impact.

2.2 Early Argumentation Theory in AI

Argumentation diagrams have boosted the entire field of computational linguistics, but as they are theoretical models, they do not have the capacity to reform the entire argumentation field. In the 1990’s, the first research connecting argumentation with the area of Artificial Intelligence (AI) was conducted. These early attempts on connecting arguments with AI created a new field under the name Computational Argumentation, the field which has formed AM in a great extent. Computational Argumentation is used extensively in domains where reasoning process is sophisticated, such as law or medicine, and therefore traditional methods like formal logic and classical decision theory cannot be applied.

One of the first researches studying the relationship between argumentation, cognitive psychology and AI was conducted by Pollock [52] describing the connection of defeasible reasoning in philosophy and nonmonotonic reasoning in AI through the definition of a set of rules. An important step ahead was made by Dung [53], who researched the relation between argumentation and Logic Programming, focusing on the modelling of the fundamental mechanisms humans use in argumentation and expressing them with a number of rules and definitions capable to be interpreted by AI algorithms.

Similar to Dung’s method, Krause et al. [54] developed the logic of argumentation, an approach for defining reasoning in cases of uncertainty. By defining rules, definitions and propositions they have managed to create a theoretical system capable to estimate the strength of an argument based on the distinctive axioms used for its construction. The degree of justification was also researched by Pollock [55], focusing on the different degrees of justification the distinctive components of an argument can provide when they are ”summed”.

Another work that indicates the relationship between argumentation and logic programming was held by Parsons et al. [56], where a framework was developed capable to interpret a broad range of negotiation in a multi-agent system. The distinctive arguments of the agents are considered as logical steps towards to an acceptable compromise, not necessarily towards the optimum proposal. A more detailed review on the automated negotiation is held in [57] presenting a more generic framework and three different approaches (game theoretic models, heuristics, argumentation-based) for the negotiation process. The development of a dialectical argumentation scheme was introduced in [58], where the authors implemented a fully-functional agent capable to maintain the natural flow on the debate by arguing on a topic and in the same time responding to obligations related to the discourse.

Although many of the aforementioned frameworks are designed to cover argumentation in its full generality, any system with pre-defined rules is not suggested for use in the open and constantly changing environment of social media, as it does not have the ability to learn and adopt. In Table 1, a synopsis of the theories and schemes that have been presented in this section are illustrated and evaluated based on a number of criteria focusing on their suitability on AM in social media, but the table could could also be used as a point of reference for tasks in the wider area of NLP.

The first criteria of a theory’s evaluation is the explicit expression of any kind of relations between the distinctive components of the argument, a property that can facilitate advanced NLP tasks such as relations identification, evidence detection, and facts recognition. In the theories that are based on logic programming, the relations are expressed as rules, whereas in Toulmin’s theory is expressed through the qualifier and in Beardley’s as modality. The second criteria in the comparison table is the level of complexity, which is determined based on the number of components and relations each theory involves. Both the in-depth analysis and the construction of the reverse tree in Whately’s theory, and the definition of six different components in Toulmin’s diagram present as expected high levels of complexity. Once again logic programming theories can be grouped together presenting high complexity as they define multiple rules and axioms in order to cover a wide range of cases. A special note should be given in RST, as it is a flexible open scheme, where components and relations can be defined and modified based on each case study. The first assessment of the theories/schemes take place based on their suitability to be applied in NLP tasks in noisy environment. The complexity level of a theory is inversely proportional to its suitability when applied on noisy environment as complicated reasoning processes cannot be deployed in text lacking grammatical and structural rules. Finally, the last column of the table illustrates the applicability level of the theories for AM tasks independently from the source of the content, where three schemes stand out (Beardsley, Toulmin, RST), each one for different reasons. Beardsley’s theory is straightforward enough to be applied easily in a wide range of goals, Toulmin’s detailed scheme is the option for in-depth analysis and RST can be easily modified in order to cover the needs and the requirements of each case study.

Computational argumentation contributed significantly in the creation of the AM field, but the strict rules that used to be posed seem outdated. Although the foundations of logic programming are based on knowledge- and logic-based solutions, the recent advances in ML have been used extensively in data-driven approaches [59, 60], but they seem to have reached the upper limit of their capabilities. Rule-based machine learning could be considered it as a data-driven approach of inductive logic programming that combines background knowledge and ability to learn based on human readable theories. Another approach combining a data-driven solution in unison with argumentation structure took place in [51] where the probabilistic classifiers have been improved with the incorporation of an argument database.

3 The status of AM in Social Media

AM is considered a complicated task because it challenges a series of closely interrelated distinctive tasks that come under this general term. The classification of arguments consists from a series of steps and it is considered the final output of the pipeline, however, other distinctive parts of the pipeline, such as argument detection, could form the main research question. Especially in the world of social media (and web-generated context in general) tasks that are focused on the reliability and the strength of the arguments seem to gain the interest of the research community. In the rapidly changing environment of social media, the presence or not of argumentative features can assist scholars in the tasks of rumour spreading, fake news detection and source identification.

A recent survey [37] suggests a general-purpose pipeline for the task of AM with two main components, argument component detection and argument structure prediction. The proposed architecture can be generalized in every source of raw text without significant problems, however if it has to be applied in AM tasks from text derived from social media would face some obstacles, as either important generalizations would take place either some tasks would be ignored. In our effort to discover better the connection between AM and social media we describe the basic tasks that take place in an AM pipeline and compare the results gained from different research methods. In 3.1 the task of argument detection is presented, followed by the tasks of relations identification in 3.2 and stance detection in 3.3. In the last sub-section, different tasks that are related with the concept of argument reliability in social media are presented.

3.1 Argument Detection

The classification of a sentence or a series of sentences as argumentative or non-argumentative is a crucial step towards AM in social media. Argument detection is in essence a preliminary binary classification that would enable the subsequent in-depth analysis of the argument, such as persuasiveness detection, relations identification between the components of the argument or automatic evaluation of the argument.

Data from social media are a characterized as a special category of web-generated data and this becomes clear if an attempt is made on applying the Toulmin scheme in a typical tweet. In the work of Addawood and Bashir [3] the following tweet from their dataset is presented: RT @ItIsAMovement ”Without strong encryption, you will be spied on systematically by lots of people” - Whitfield Diffie. The above tweet cannot easily fit in the Toulmin’s scheme (or any other theoretical scheme) as the fact consists the same time the conclusion of the argument, whereas the component of the backing is expressed through the quote of an expert opinion. A similar belief is also expressed in [61] where the authors claim ”we (almost) never find such a kind of complete structure of the arguments”.

Towards to the analysis of the tweet, the first step is defining the necessary background knowledge that a reader/annotator has on the discussed topic, in order the IAA not to be affected by the different level of the annotators’ knowledge. For the ease of the annotation task some general rules can be defined that can provide a useful insight for the nature of the text. When the user cites the opinion of an expert, express a feeling or irony it is very likely the tweet to be argumentative. On the other hand, if the tweet is the title of an article and simply shares a link without any comments, the tweet can be characterized as non-argumentative. An example of a non-argumentative tweet is provided in the work of Dusmanu et al. [63] regarding Brexit: 72% of people who identified as “English” supported #Brexit (while no majority among those identifying as “British”) https://t.co/MuUXqncUBe.

The rules that have been described should be consider more as guidelines, as there are many cases where a tweet cannot easily fit any of the proposed categories. In many cases a twitter user takes into account more information than simply the information contained in the tweet, as the beliefs or the status of the user who tweeted. Thus, defining a specific level of knowledge of the annotators is crucial for every use case.

Other important factors that affect the annotation process and the quality of the dataset are the requested task, the size of the dataset, the debate topic and the number of the annotators that have been used for carrying out the specific task. The above aspects can be used as metrics for measuring the quality of the collected data, as often scholars put thresholds in IAA for accepting or rejecting specific datasets. Table 2 provides a detailed review of the characteristics of the datasets that have been used in recent research papers and are analysed in the following subsections. The table includes both annotated datasets that have been used in supervised ML approaches, but also datasets that have been used in un-supervised approaches. There is a scarce of annotated datasets in AM and most importantly there is a difficulty in their re-use, as they are often annotated for very specific tasks. The recent review paper of [38] provides a complete synopsis of the available datasets in the area of AM, without focusing on social media generated text.

Based on the results depicted in Table 2, the IAA depends heavily from the task at hand as it is revealed in Addawood et al. [60] and in Addawood and Bashir [3]. On those two research papers, the same annotators, in the same dataset present higher IAA for the tasks of Evidence/Topic Classification in comparison to the tasks of Argument/Stance Detection, although the later offer more possible classes from the former. In the case of limiting the possible classes for the same task, as in [74], the IAA increases as it is expected, but the task ceases to provide a detailed analysis. The last point that should be stressed considering the IAA is the different metrics that are used in the different studies, as the number of annotators affects the available options.

The task of argument detection as a necessary preliminary step takes place in the work of Addawood and Bashir [3] and also in Dusmanu et al. [63], where the proposed pipelines end up in the recognition of evidence type and in the source identification respectively. In both works the detection of argumentative tweets is necessary, as the non-argumentative tweets consist respectively the 42.3% and the 29.3% of the constructed datasets. Both papers adopt a supervised approach where a manual annotation is required, succeeding a substantial inter-annotation agreement (IAA) in terms of Cohen’s kappa equal to 0.67 and 0.77, respectively. A similar architecture was also adopted in [61] ending up in the construction of argumentation graphs. The DART dataset [79] was used as input of the pipeline, containing 2702 argumentative tweets and 1181 non-argumentative tweets. In contrast to [3, 63], Bosc et al. [61] measured the IAA in terms of Krippendorff’s alpha, resulting to the satisfactory a=0.81.

On the other hand, a semi-supervised approach was followed in the papers presented in CLEF 2018 conference for the task of Multilingual Cultural Mining and Retrieval. In the work of Deturck et al. [62] the assumption that an argumentative text is structured to effectively combine arguments and opinions takes place, thus argumentation is measured through structuration. A similar approach was followed in Sendi and Latiri [64], where argumentative tweets are defined as the sum of three separate tasks: information retrieval, topic modeling and sentiment score. The work from Dufour et al. [65] follows a distance supervision approach through the detection of five pre-defined features: emotion words, emoticons, particular punctuation signs, personal pronouns and hashtags.

The features used in the aforementioned research, but also for those that they are going to be presented in the next sections, are provided in Table 3, where are summarized in 5 different categories. Lexical features are the attributes that are used most frequently in the wider spectrum of the NLP and they are strongly correlated with the different applications of n-grams, whereas as semantic features we define the characteristics of the language that can provide a deeper insight of the data. Sentiment features are those who can trigger emotions and usually they are detected with the use of specific lexicons or libraries and the subjectivity features often indicate an opinionated and therefore an argumentative tweet. The twitter-specified features are offered as metadata through the twitter API and concern the specific characteristics a tweet contains, and in the last column we have collected the features that they cannot be grouped under any of the previous categories. Apart from the semantic and sentiment features, LIWC offer statistics that include personal concerns, core drives and needs, which are summarized under the psychometric category. In the work of Deturck et al. [62] the use of word embeddings takes place aiming at a diversity filtering in order the most argumentative tweets to be discovered.

It has to be mentioned that the classification of features is different in [3] (the only work providing a detailed table of features), classifying emotional tone and subjectivity score under the linguistics features. Furthermore, we include the tasks of information retrieval and topic modeling as described in [64] as lexical feature, provided that they do not discover any semantic purpose. Those adjustments took place in order to provide a useful taxonomy and a beneficial comparison, but it should be noted that different categorizations can take place.

Regarding the results of the classification, those can be extracted from the works that have followed a supervised approach and they are presented in Table 4. In the first column the names of the authors are depicted, in the second and the third columns the algorithms and the features that have been used are displayed respectively. The last three columns depict the metrics that can evaluate the performance of the algorithms. Concerning the feature selection and their impact for the classification task, the use of all the possible features performs better in each case. The selection of the classification algorithm does not seem to affect significantly the performance of the task in [63], whereas in [3] the use of SVM surpasses the alternative algorithms. For the task of argument detection, the best results are achieved in [60] with 0.89 F1, whereas [63] and [61] achieve 0.78 F1 and 0.67 F1 respectively. The results of the semi-supervised approaches are measured in terms of ranking quality either using the Normalized Discounted Cumulative Gain (NDCG) either simply presenting qualitative results.

3.2 Relations Identification

The choice and the customization of the theoretical argumentation model that will be adopted in any research project affects the individual tasks that will be raised. Especially the task of relations identification, or argument structure prediction as expressed in [37], is the most susceptible part of the AM pipeline to potential changes in the adopted model. The task of annotating relations between parts of text requires the adoption of a holistic approach, capable to identify connections with both preceding and succeeding components of the selected model, the relations between the entities of the network and eventually offer a better understanding of the argument.

Both Toulmin’s [45] and Freeman’s [48] theories, two of the most influential theories in the wider field of logic and argumentation, explicitly define relations between the components of the arguments. In data derived from social media, argument’s component identification is a challenging task as both their size and their chaotic nature do not allow strict rules and principles to be applied. As a consequence of this situation, the task of relations identification should be redefined and include both micro and macro analysis. The micro-analysis evaluates the quality and the completeness of the argument, whereas the macro-analysis expresses the relation of an argument either towards a known topic or towards an argument previously expressed. In social media, network analysis algorithms can significantly boost macro-analysis tasks, as the introduction of network-based features reveal underlying relations between the users and eventually improve the prediction model [72].

The possible outcomes of a macro-analysis in social media text are limited to support/attack/neither relations indicating in a great extent the outcome of the stance detection. On the other hand, micro-analysis is related more with AM and other reliability-related tasks, as it evaluates the integrity and the cohesion of the argument. The arguments extracted from online resources are not characterized as high-quality data, as often complicated reasoning process should take place in order the argument to be understood. Arguments with missing premises are called enthymemes [8, 9] and take place often in informal discourse, creating a challenge for the approach that should be followed; discard the argument or try to fill the missing premise.

The need for both micro and macro analysis for the task of relations identification, in combination with the low-quality data from social media creates the need for the establishment of simple, but effective rules and standards. Towards the need for providing a straightforward definition able to capture both micro and macro analysis, we define three entities (argument, topic, completeness) able to capture the complicated nature of the task. We convert the problem to a mathematical expression in a triple, where the task of relations’ identifications is split in two parts, where the first part connects the argument with a specific problem (favor/against/neither) and the second part evaluates the structure of the argument. Eventually, expressing the relations identification as a triple we have:

[TABLE]

where the expressed argument aij is open to a macro-analysis considering a topic ti and a micro-analysis for its completeness cj.

Only few researchers have explored the task of relations identification in text derived from social media, because of the chaotic nature of social media and the wide presence of vague claims. In subsection 3.1 the first step (argument detection) in the proposed pipeline of Bosc et al. [79] was presented, which is followed by the prediction of attack/support relations between tweets and arguments. Their adopted approach is similar to textual entailment, thus the Excitement Open Platform (EOP) and the Recognizing Textual Entailment (RTE) framework were used. A second method was also applied implementing a neural sequence classifier, however none of the methods presented satisfactory results. In fact, the detection of support-relation achieved 0.20 F1 and the attack-relation 0.16 F1 using neural model, and with use of EOP+RTE the support-relation achieved 0.17 F1 and the attack-relation 0.0 F1. Apart from the low score in the automatic detection of relations, even the IAA was significantly lower a=0.67 for the specific task, comparing to IAA for the task of argument detection which reached a=0.81.

A new method for extracting argumentative relations of attack, support or neither is presented in [66] based on the Relation-based Argumentation Mining (RbAM) model. The proposed model was tested in the dataset of Carstens and Toni [51] and afterwards it was applied on the task of relations prediction between tweets and news headlines on two different datasets [82] [83]. Apart from different implementations of neural networks, the impact of trained and non-trained was also evaluated demonstrating the supremacy of the trained embeddings. Apart from the use of word embeddings and the argumentativeness features extracted from RbAM, the authors do not describe the rest of the features, instead they simply use the term standard features, thus safe conclusions for the use of features cannot be drawn.

Broadening the limits of relations prediction task, Ma et al. [67] introduced a 3-step framework including both micro and macro-analysis of the argument, in contrast to [79] and [66] where only a macro analysis took place. Besides the attack/support relation between a tweet and a topic, the authors also examined the relatedness of a topic towards the pre-defined topic and the existence or not of an arguable reason, where the evaluation of the argument toward its completeness take place. Considering the complexity of the proposed methodology and the comparison with state-of-the-art baselines [68, 84, 85], the presented results that can be characterized as promising. The entire process is characterized as information retrieval task, thus the learning-to-rank approach was adopted and the metric precision at k was used, which indicates the precision among the k top results of the retrieval. Considering the three distinctive sub-tasks that consist the claim-oriented tweet retrieval task have an increasing complexity, as it is pointed out through the report of the IAA, where topic-relevance reached 90.1%, clear stance 78.2% and detection of arguable reason 75.2%.

In Table 5, a summarization of research papers on the task of relations identification in text derived from social media takes place. The first column of the table presents the authors of the paper, the second one interprets the scope of the task(s) according to the proposed definition and the task are presented in the third column. The next three columns present the technical details (algorithms, metric, score) for the implementation of each proposed method. In [66] the results for the two datasets that their proposed methodology has been tested are presented. It should be stressed that the scores are not directly comparable, as different research papers carry out different tasks; instead we should focus on the coexistence of different approaches and the level of difficulty of each one.

Each one of the presented research papers exploits data derived from twitter, as the focus of this literature review paper is AM in social media. Web-derived text seems to thrive as a source of AM pipelines, including the task of relation identification, but the source of data usually is a more structured source of text, such as debate forums [86, 87, 88, 89]. Although the information found in social media are characterized as noisy text and it is far from an ideal scenario for AM, the constant generation of content allows to the researchers to conduct research including the time axis in order to understand users’ behaviour [72, 90] and evaluate their impact beyond the network [91, 92]. Users in social media platforms usually express emotions or quick messages with very little argumentation, however the introduction of argumentative features can enhance other NLP tasks [60, 66]. Both micro [93] and macro [94] analysis have the attention of the research community, whereas they have been approaches that combine them [87, 88]. Another research topic that has gained the interest of the research community is the reconstruction of implicit warrants, although the existing research papers [8, 9, 29] do not utilize social media as source.

3.3 Stance Detection

In contrast to relations’ identification task, stance detection is a popular task among researchers in the NLP community, either as autonomous and independent task either as a part of an extensive pipeline. Stance detection is thriving even in the challenging environment of social media and Twitter is often used as a source of information.

Stance detection is related to many sub-fields of the wider NLP area, such as sentiment analysis, textual entailment, topic extraction and AM. In the context of AM in social media, we define stance detection as the task of automatically determining the attitude of the author towards a specific topic exploiting any kind of information that can be collected. The stance can be determined either exclusively by the content of the text either from combination of features that are capable to reveal specific characteristics such as argumentativeness [60] or network communities [72, 95].

The main difference between opinion mining and stance detection as it is expressed in AM pipelines lies in the concept of data aggregation from the wider environment towards the final outcome. Stance detection is considered as the final part of the pipeline that exploits the findings of the previous steps, rather than a stand-alone task.

The main difference between opinion mining and stance detection as it is expressed in AM pipelines lies in the concept of data aggregation from the wider environment towards the final outcome. The term wider environment applies on both combination of sources and tasks [96], as web-generated data and especially social media offer an excellent environment for sentiment analysis, but a poor one for argumentation or opinion mining, thus stance detection is considered as the final part of the pipeline that exploits the findings of the previous steps, rather than a stand-alone task.

As the research community has shown great interest in the task of stance detection, it is impossible to present each research paper, rather we decided to focus on the research methodologies that either aggregate data from different sources either have the ability to be used as part of a bigger system. Similar to relations’ identification task, we also provide a mathematical expression of the stance detection, influenced by [97] on the definition of opinion mining. Stance detection is expressed as the quintuple:

[TABLE]

where hi is the person who holds a specific stance sijkl for a specific debate dl, justified by a rationale rk in a specific time tl. The rationale of the stance for a specific debate can be assessed for its quality [63, 3] through a variety of sub-tasks, such as facts identification, evidence recognition, source classification and reasoning evaluation.

The sixth task of SemEval-2016 [99] introduced the shared task of stance detection in tweets, providing a significant boost in the field as new methodologies were suggested and the constructed dataset was also used in later research. The shared task consisted of two parts regarding the supervision framework to be followed (fully-supervised, weakly-supervised). As the constructed dataset was used afterwards from the completion of the task, more modern approaches have surpassed the top performances described in the task.

The best-performing system of the competition [69] proposed a recurrent neural network capable to extract information from unlabeled datasets using word embeddings. The use of word embeddings as features was also critical in [68], where a simpler linear SVM algorithm achieved F-score up to 0.70, surpassing the previously highest score of 0.68. Apart from the use of word embeddings, the presence or absence of the target of interest in the tweet improved the results of the algorithm. One more improvement in the same dataset for the same task was achieved by Wei et al. [70] reaching the F1 to 0.71, where an end-to-end neural model was proposed which makes better use of target information.

Considering the results for the weakly-supervised framework as described in [99], those are significantly lower as no training data are provided for the topic that is researched, thus the developed methodologies rely heavily on techniques that can transfer knowledge from different topics. The submission with the highest performance for the task achieved 0.56 F1 [98] and proposed a convolutional neural network including a modified softmax layer able to perform three class classification, although the training is consisted of two classes. A weakly-supervised approach exploiting the network structure information proposed from Ebrahimi et al. [71] improved the previously best score and reached 0.57 F1.

Weakly supervised approaches exploiting social media features for political stance detection were also adopted in Lai et al. [72] and in Johnson and Goldwasser [73]. The former is focused on the Italian referendum in 2016 and employs a holistic approach for the stance detection adopting a diachronic perspective for the user’s stance including twitter-specific features and social network communities, achieving 0.90 f-micro with the use of svm. A similar approach was also followed in [73], able to capture both the content and the social context through linguistic patterns reaching 0.86 accuracy. The novelty in their approach lies on the absence of manual annotation, as the annotation of the political stances took place with the use of ISideWith.com.

In the work of Addawood et al. [60], an advancement of a previously established scheme [3] is proposed capable to carry the task of stance detection resulting in a F1 score of 0.93 with the use of the Decision Tree algorithm. In this paper, the introduction of argumentativeness as a feature take place increasing significantly the performance of the algorithm. The findings of the paper indicate that argumentativeness features are the most informative ones for the successful categorization of favor and neutral categories, stressing the importance for introducing AM techniques in different text mining tasks.

Table 6 presents the algorithms, the metrics and the score selected research papers have achieved. In the second column the possible classes that are provided to the classification algorithms are depicted, there is either a binary approach (favor, against) either the ‘neither’ option is included. Considering the algorithms that are used, SVM and neural networks algorithms are used, apart from [73] where a probabilistic soft logic was adopted. In the last two columns, the score that has achieved measured with different metrics are presented. As it is expected fully supervision ML approaches achieve higher results when they are compared to weakly supervision ML approaches, when both applied on the same dataset. Apart from the ML approach, defining the number of possible classes plays a role in the performance of the algorithms, as the binary approach achieves normally higher results.

3.4 Reliability-related tasks

The wide use of Twitter in combination with its public nature have established it as the most appropriate social network for studying a variety of tasks related to virality, such as viral marketing, rumour diffusion, event and fake news detection. The majority of the proposed methods are relied to social network analysis, exploiting the metadata offered by the social network (friends, followers, time of publishing, etc.). As the scope of this paper is neither an in-depth review of rumour detection techniques and methods, nor the evidence identification for claims in any kind of text, we are going to focus on research work that connects argumentativeness with the evaluation of the argument’s reliability in text derived from social media.

The Twitter platform is often used as a mean of expressing arguments for controversial topics, some of them are efficiently supported with facts and evidence from reliable sources, whereas in some other cases instead of backing their claims, they simply express feelings or unsupported allegations. The constant data generation and rapid pace of news flow create a chaotic environment with limited time for claims to evaluated and facts to be assessed. Due to the environment that has been created, where users express opinions and views in real time without using sophisticated or pretentious vocabulary, a unique opportunity is raised for argument evaluation on various political and social issues. The automatic evaluation of arguments has the potential to reduce the incidents of rumour spreading, the faster detection of fake news and eventually the improvement of the quality of public political discourse.

An essential part of a complete argument is the sufficient backing of the original claim, either in the form of premises either in the form of backing with evidence and facts presentation. A simple, but robust structure of claim and supporting evidence is adopted in [3] for the classification of arguments’ evidence, where the ultimate step of the proposed pipeline is the classification of evidence to six different categories. The proposed pipeline of Dusmanu et al. [63] contains two tasks that are related to the reliability of the expressed argument, the distinction of factual information from opinions and the source identification from a pre-defined list. For both the tasks the use of all the available features boost the results of the classification.

In the work of Konstantinovskiy et al. [74], where the objective is the construction of a reliable fact-checking mechanism, both an annotation scheme and an automated claim detection method is proposed. Two different approaches are presented, the binary model of claim/no claim and a multi-class classification with seven categories describing the claim. The proposed methodology (the binary model) overcomes previously established mechanisms (Claimbuster, ClaimRank) in terms of F1, as it achieves F1 0.83, while the multi-class classification displays the impressive 0.70 F1-micro, and in terms of macro average it achieves 0.48 F1-macro.

In Table 7 a summarization of the research work that accomplishes reliability-related tasks in the context of AM in text derived from social media is presented. Four tasks have been recognized in this category and the results of the different approaches are heavily relied on the number of the alternative classes. The 0.83 F1-macro that is achieved in [3] is impressive if we consider that there are six available classes, and for a similar task with seven available classes the F1-macro is 0.48 in [74]. The difference could be merely explained through the exploitation of more features in [3] in comparison to [74], where not that advanced features were used.

Besides the aforementioned research, there have been some approaches on connected reliability and evidence with argument’s strength, but they are not applied in social media text. For example, a research work that uses argumentativeness as a feature, apart from [60], takes place in Cocarascu and Toni [66] for the task of deceptive reviews detection, leading to an improvement of the prediction algorithms. It has to be noticed that the exclusive use of argumentative features without topic modelling or the use of additional features cannot surpass the baseline. The work of Park and Cardie [6, 7] is another great example of combining argumentation with evidence classification, suggesting three different categories of justifications in online user comments, but as in [66], both papers exploit more structured forms of arguments. Similarly, the task of context dependent claim detection [100, 101] utilizes hundreds of Wikipedia articles, but it has not been applied in social media text.

Another task that is related with both the quality evaluation of the argument and the in-depth analysis in macro-scale is the enthymeme reconstruction. Although the task has not been applied yet in data derived from social media, at least in our knowledge, it has been tested in web-generated data [8, 9] and we strongly believe that text from social media offer an excellent testing ground for the identification of implicit supporting claims. The task of reasoning comprehension as it has been presented in [29] has the potential to be applied in social media and advance the wider field of AM, but the complexity of the proposed methods raises concerns for the transferability of the task to new environment and un-trained annotators.

4 Existing Tools in AM

The increasing interest in AM has increased the need for suitable tools, such as grammar parsers, sentiment lexicons, software for boosting the manual annotation tasks, and tools capable to automatically extract arguments from natural language. In the area of NLP, there is a wide range of available tools covering various aspects and addressing different challenges. However, there is lack of standardization, accessibility, and acceptability of the existing tools, due to proprietary formats developed for modeling of natural language in different domains.

In subsection 4.1, we provide a synopsis of some of the most prominent tools in the area of NLP and how they could improve the process of accomplishing AM tasks. In the second subsection 4.2, we narrow down the area of interest and focus on tools that are specified for the task of AM.

4.1 General-purpose NLP Tools

The annotation process is of major importance in any NLP system, thus different tools have been proposed following different approaches. The first introduced web-based opensource annotation tool is BRAT 111http://brat.nlplab.org/index.html [102], which is based on the STAV text annotation visualizer [103] and is characterized by the wide variety of tasks that can be accomplished and scientific work that has been conducted using it. It has been adopted in different fields like visualization, entity mention detection, event extraction, coreference resolution, normalization, chunking, dependency syntax, and meta-knowledge annotation.

A tool following the approach of BRAT is WebAnno [104] which has kept the web interface and visualization capabilities of BRAT and modified the server layer. WebAnno has improved specific weaknesses of BRAT focusing on user and quality management with the addition of monitoring tools and interfaces for crowdsourcing. Currently, WebAnno is in version 3.0 and also offers a web-instance222https://webanno.sfs.uni-tuebingen.de/ through the CLARIN-D infrastructure. Both BRAT and WebAnno are an open and live project that could easily be modified in order to include tasks in the sphere of AM, such as argument detection, relation identification or reasoning evaluation.

The construction of graphs for text annotation is followed in GraPAT [105] covering different tasks like sentiment analysis, argumentation structure, rhetorical text structure, and natural visualization of the annotation process. The initial goal for the development of GraPAT was to increase IAA and maximize the automation of annotation without neglecting neither the variability nor the annotation speed/comfort. GraPAT can be considered as the successor of RSTTool [106], as it maintains the principles of RST annotation, enriching it with more capabilities like sentiment analysis and argument structure annotation model.

GATE [107] has dominant presence in the wider field of text engineering offering numerous tools and capabilities from simple tasks (e.g. information extraction, named-entity, etc.), to modifications for cutting-edge technologies, such as cloud-enabled software and social media analysis. Regarding the argument annotation task, GATE offers Teamware [108], a web-based software suite which provides the environment for collaborative annotation and curation. GATE Teamware stands out as the only annotation tool, to the best of our knowledge, which supports execution of an automatic NLP system, before manual annotation.

The DiGAT tool [109] has been developed alongside with an annotation scheme and a graph-based inter-annotator agreement measure based on semantic similarity. Similarly to GraPAT, DiGAT also relies on graph structures for the annotation process, aiming at simple and accurate annotation of relations between entities in long text.

The establishment of the TextCoop platform alongside the Dislog language is presented in [110]. TextCoop is the only tool in this subsection that follows a logic-based approach, heavily influenced by RST, modeling the conclusion as a nucleus and the support as a satellite. As it is described, TextCoop offers a functional web interface, however, to the best of our knowledge, this is not provided, yet. Another tool that is in a similar status with TextCoop is TURKSENT [111], a manual annotation tool with multi-lingually capabilities aiming at automatic sentiment analysis of text derived from social media. Its mentioned web-based interface does not seem to be available, yet.

The Stanford CoreNLP toolkit [119] receives great acceptability from the NLP community as it offers a broad range of grammatical analysis tools, different APIs for the majority of the programming languages, and the ability to run as a simple web service. However, a specific tool for AM-related tasks is not yet developed. Among the existing tools offered by the toolkit, the Stanford OpenIE is more closely related with AM, as it enables the extraction of relation tuples out of binary relations. Two more tools included in the toolkit and can be used in an AM pipeline is the Stanford Relation Extractor, which finds relations between two entities located by the Stanford Named Entity Recognizer, and the Neural Network Dependency Parser, a dependency parser that establishes relationships between ”head” words and ”modifier” words in a sentence. The web interface provided by the toolkit333http://corenlp.stanford.edu/ offers up to ten different annotators and the visualization of the schemes has been realized using the BRAT software.

The majority of the aforementioned tools are on-going, open-source projects that can be modified in order to carry out AM-related tasks, whereas some others (GraPAT, DiGAT) are graph-based annotation tools that are used for identifying the relations between chunks of text. Other functionalities such as sentiment evaluation and name entity recognition can boost AM related tasks, as sentiment features are used in the majority of the existing research papers in the area as Table 3 illustrates. If any of the subtasks that are included in the AM pipeline can be executed automated and reliable through a software tool, then this tool should be exploited.

4.2 Argument search, retrieval and automatic annotation

In this sub-section, we present the tools that have been designed to enhance the argumentation process. Some of those tools have been implemented to boost the annotation step, others offer an argumentation search engine and there are tools capable to automatically grade an argument or even perform the entire process of AM.

The Centre for Argument Technology444http://www.arg-tech.org/ has produced a series of tools covering different aspects of AM. The latest developed tool is OVA555http://ova.arg-tech.org/ [114] and has replaced to a certain extent the tool of Araucaria [120]. Arivina666http://arvina.arg-tech.org/ [121], the successor of MAgtALO [122], offers a dialogue system implementing the concept of mixed initiative argumentation, where human players and agents debate having equal levels of participation. The process for the calculation of the acceptability semantics on structured argumentation frameworks is completed through TOAST777http://toast.arg-tech.org/ [115].

Probably the most influential achievement of the considered organization is the establishment of the Argument Web888http://www.argumentinterchange.org/ [123], a repository in cooperation with a series of tools, systems and services, such as AIFdb, the main search interface for the Argument Web. ArguBlogging999http://www.argublogging.com/ [124] materializes the concept of crowdsourcing in argumentation by capturing arguments that take place in online platforms (tumblr and Blogger are supported) and provides them as feed to the Argumentat Web. Concerning the educational aspect of argumentation, Argugrader101010http://www.argugrader.com/ offers automatic grading and provides detailed feedback to students regarding successful or not construction of arguments.

A prototype argument search framework is proposed in [116] able to carry out the entire process of an argumentation search engine, from user query and argument retrieval to ranking and presentation of arguments111111http://www.arguana.com/args. The argument search engine is relied on pre-structured arguments from a defined list of debate portals, and a standard mapping process takes place in order to convert the concepts that characterize each argument in the different debate portals to the common argument model.

A system able to utilize the heterogeneity and big volume data is implemented in Stab et al. [117], under the name ArgumenText. The ArgumenText uses 400 million plain-text documents from different sources and deploys a series of technologies in order to construct a solid pipeline able to materialize a sequence of sub-tasks that eventually present ranked pro and con arguments through a web interface.

At this moment we are aware of only one tool that accomplishes the complete task of automatic annotation in terms of AM. MARGOT [113] is built on the foundations of Lippi and Torroni [125] and extends the previous established model by including the task of evidence detection and providing a web interface121212http://margot.disi.unibo.it/. The syntactic structures that are followed in argumentative discourse is the fundamental idea on which the tool was built. The model implemented for AM involves two binary classification problems, the argumentative sentence detection and the argument components boundaries detection. The former is addressed with a combination of tree kernel and bag-of-words, whereas for the latter SVM-HMM with bag-of-words, part-of-speech, lemma and named entity are employed. In overall, the tool achieves acceptable evaluation scores, regarding the complexity of the task, but there is still room for improvement in covering various domains.

Another tool that is very similar to MARGOT, sharing the same interface131313http://155.185.228.137/claudette/, is CLAUDETTE [14], an on-line platform addressing possible unfair or abusive terms of service. The platform realizes a 2-step algorithm including the binary task of detecting an unfair clause and if the first step is positive, then a classification task through 8 possible categories takes place. For this classification, a combination of eight SVMs exploiting lexical features is used with fair results in all used metrics.

On industry level, Austhink Pty Ltd. is continuously developing software tools aiming at the improvement of the general reasoning and argumentation process through training sessions. The two most successful software tools are Rationale [118, 126] and bCisive [127], with both of them offering online limited free versions141414https://www.rationaleonline.com/editor/151515https://www.bcisiveonline.com/editor/. The former supports a series of activities in the area of AM, such as construction, visualization and mapping of arguments, while the latter focuses on providing support to business decision through hypothesis and decision maps. Rationale is considered as the successor of Reason!Able [128].

A synopsis of the existing NLP tools that have been discussed in this section is presented in Table 8. The existing tools can be classified into three categories, tools that aid the manual annotation process, general-purpose NLP tools and tools that offer an entire mechanism for argument search, retrieval and evaluation. It has to be underlined the evaluation of the argument differs to the different approaches, as in [113] the number of claims and premises are presented, in [115] the weight of the argument is calculated and in [116, 117] the arguments are categorized as pro and con.

5 Proposed conceptual framework and future directions

As every scientific field that is in a premature level, AM has to take steps towards its establishment in the wider research community of NLP by invoking the interest to a greater audience through more applications of AM in a wider spectrum of scenarios. Towards this vision, there are specific challenges and concerns that have to be faced and technologies that have to be tested. In the remainder of this section, we propose a conceptual framework able to capture the needs of AM in social media and boost NLP-related tasks, and present some promising approaches that can significantly contribute in the AM process. Our proposed conceptual framework is in our knowledge the first pipeline capable to connect various NLP tasks between them having in prominent position the AM tasks in noisy environment. Previously proposed pipelines are either focused on the task-at-hand [3, 129, 6] either do not present any connection with other NLP-related tasks [37].

5.1 Proposed conceptual framework for AM in social media

The two most dominant characteristics of text derived from social media is the short length and the lack of defined norms. Considering the fact that the typical length of a tweet is less than 50 characters, the definition of argument boundaries in many cases is not feasible, as an argumentative tweet can hardly contain information unrelated to the major claim. On the other hand, the already unstructured nature of text data in combination with the massive use of jargon and emoticons establish an environment where any lexical rule is really challenging to be applied, as either has to be specific for each case study either loosely defined in order to include different cases. Both assumptions would probably lead to lack of transferable knowledge, a crucial objective for almost every proposed methodology.

A possible criticism for the lack of the boundaries-definition task as defined in [37] is the increase of the upper limit in Twitter to 280 characters from 140, as well as the existence of various social media including forums such createdebate.com where the norm indicates well-structured arguments and length significant longer than 50 characters. However, the great dominance of Twitter in socio-political issues, which is also boosted from the online presence of political leaders, combined with the shrinking of general-purpose forums have formed an environment which the use of social media data seems to be the only source of web-generated data in the near future.

The increase use of social media in the wider area of AM as a source of information highlights the need for the definition of a new scheme devoted for the specialized procedures in the field of AM in social media. The proposed conceptual architecture, as it is depicted in Figure 5, contains three main components and each one includes distinct tasks that could form a second focused pipeline, but could also exist independently from the other tasks. The first component contains the core tasks of an AM procedure, the tasks that make the heart of the pipeline and can be applied in different text data, from official political speeches to tweets and comments in products or services reviews. The other two components include tasks that are involved somehow in the process of evaluation the reliability of the text. The additional components should not be underestimated and should be considered of equal importances compared to the AM core task, as the interest of the scientific community is constantly increasing for tasks such as fact checking, evidence and fake news detection, even though the connection between AM and reliability-related tasks is not yet very solid.

The first step of the first component in the proposed conceptual architecture is the task of argument detection, where the identification of a sentence as argumentative or not takes place. A significant amount of work intentionally ignores the step of argument detection [26, 72], as argumentative rhetoric is a prerequisite for the evaluation of persuasive essays or political debates. When collecting heterogeneous data from social media the detection of argumentative text is unavoidable, as not all users intend to persuade for or against a discussed topic, but simply express a reflection, a feeling or a question. The task of argument detection is considered as an essential step for the AM pipeline in social media, as the following steps of pipeline is not possible to be completed if is missing.

A great amount of research work is focused on the identification of the components that construct the argument, especially the early attempts of argumentation (see section 2.1), presenting as the ultimate goal the successful analysis of the argument, emphasizing in the reasoning concept behind the structure of the argument. The concept of in-depth analysis of an argument through the identification of the argument’s components and the discovery of the underlying relations between them has been adopted and developed in the field of AM in social media, adjusting to the new environment including the relations and the interaction between the entities of the network. The two main tasks that fall under this category is the relation-based AM [51] and the enthymeme reconstruction [8]. Both tasks provide a useful insight for the structure of the argument, but instead of trying to evaluate its impact, they focus on the efficient use of small datasets [9, 66] or they are designed to aid the task of stance detection [8].

Earlier in the text we characterized Opinion Mining as the predecessor of AM, mostly because the terms stance detection and opinion mining can be used interchangeably. Stance detection is the final step of the proposed framework and it is a task that is heavily depended from the previous steps of the pipeline, as the nature of social media data demands a significant pre-process procedure. This procedure in AM pipeline is expressed through the detection of argumentative tokens and the identification of relationship between the components of the argument (explicit or implicit).

The constant generation of data in social media has raised significant concerns for the quality of information that is shared and read in social media. The connection between argument in social media and reliability has not yet discovered in depth, comparing with the amount of work that is dedicated in the areas of rumour and fake news detection. However, tasks such as evidence type classification, source identification and facts recognition have emerged in the area of AM, raising the awareness of the connection between the expressed argument and its reliability.

In the proposed conceptual architecture for AM in social media we devote a unique component of the pipeline in reliability-related tasks, in order to stress its importance and the room of the development that exists in the area. AM reliability tasks are not considered as core tasks of the AM pipeline, but they can enhance the procedure especially when applied in arguments derived from social media, as the backing of the claims is many times inaccurate or it is based on rumours or hoaxes. Other tasks that can be assisted by the progress in the field of AM and integrate parts of the proposed AM core tasks is sentiment classification [68], subjectivity/objectivity identification, topic modeling [130], and recommendation algorithms [131].

Different tasks can be easily executed and combined through the components described in our proposed architecture, as it is both detailed and easily modifiable. The example provided in 3.1 concerning the Apple/FBI encryption debate (RT @ItIsAMovement ”Without strong encryption, you will be spied on systematically by lots of people” - Whitfield Diffie), can be assessed for its argumentativeness nature, the relations that expressed through the retweet and the mention, its stance towards the discussed debate, its completeness and integrity, while other NLP-related tasks can also be enhanced from the findings of the above tasks. Our intentions on our proposed framework is to be regarded and used as a mean for enhancing various NLP tasks with the use of argumentativeness features.

5.2 Future directions: semi-supervision and background knowledge

Handling data with great volume and variety is not an easy task, especially if we consider the well-known ”gold rule”, where manual annotation is required for a subset of data. The process of manual annotation is labor intensive and time consuming, whereas possible use of unsupervised machine learning algorithms could solve the problem of lack of trained annotators. The need for novel algorithms and techniques is emphasized also in previous review papers [12, 37] and although there is an evident trend towards unsupervised [132, 133] or semi-supervised ML algorithms [7, 20, 129, 134] with notable performance, they could be further improved, as there has not been extensive work neither on the use of suitable features nor on the design of argument schemes. A recent literature review from Silva et al. [135] presents the trends in semi-supervised learning for tweet sentiment analysis and it can be used as a point of reference also in the field of AM.

Deep learning techniques are able to handle a great volume of data in an unsupervised or semi-supervised way and they have achieved break-trough results in NLP field. Deep learning has been applied [93, 39, 9] in AM, but does not seem to overpass other ML algorithms, mainly because of the limited available datasets, however more research should take place in order safe conclusion to be drawn.

In the fields of Sentiment Analysis and Opinion Mining there is a major trend towards deep learning, which is adopted in multidisciplinary fields and in various kinds of test cases [33, 136]. Concerning related applications in the AM field, several deep learning architectures are combined with unsupervised learning in the pretraining stage for feature extraction, in order to obtain important semantic features through this process. It is probably infeasible to completely solve such a complex problem, like AM, without the use of manual annotation, but these techniques could significantly reduce load from the labor intensive process of manual annotation.

The concept behind AM is finding the underlying reasoning of an expressed opinion, not only identifying if this opinion is positive or negative. In human reasoning, this process is accomplished naturally by combining a-priori knowledge towards a specific subject and knowledge on the ethos and influences of the person who express an opinion. A similar approach is followed for irony detection [137], where previous work determined that contextualizing information is required [138]. Oddly enough, an approach of that sort is yet to be followed in the field of AM, although combining multiple sources of information seems to be the natural flow of human reasoning for argument detection and classification. Exploiting background knowledge can be achieved through the use of semantic encoders mainly in models employing reinforcement learning.

The studies in the field of opinion mining and sentiment analysis are focused on real-world scenarios combining different scientific domains like politics or social networking and their roots are originated from reviews, recommendation systems, and digital marketing. Most of the research on opinion mining aims at improving specific commercial applications, whereas AM architectures have not been tested thoroughly in such scenarios. In order AM to go one step further, it is crucial that more real-world scenarios will be employed combining background knowledge and information from multiple sources.

6 Conclusion

Argumentation Mining is an attempt for a deeper understanding of natural language, is the natural evolution of opinion mining, but instead of trying to understand what others think, the focus is on understanding why. The analysis of the human reasoning process is the ultimate goal of AM and it is acquired by exploiting the inherent structure of an argument through the identification of its distinctive parts implying the inferential process that is followed.

The understanding of the human reasoning can offer unprecedented capabilities and achieve breakthrough changes in a wide spectrum of applications as information for the decision-making process can be retrieved. The existing work in the field indicates its great potential but we must consider the fact that AM is still in a premature stage and there are steps that are required for realizing human-level reasoning or at least to be able to interpret it at a sufficient level. The research community has focused on the modeling of the argument and in the suitable selection of features, whereas the selection of the ML algorithm seems not to play a crucial role for the accomplishment of the different tasks. Different approaches have been tested for modeling arguments from diagrams depictions to modifications of well-established theoretical models and no model excels in comparison with others, creating the need for establishing a flexible framework able to capture the needs of the different tasks that appear in AM problems.

In our review, we try to shed light to the Argumentation field, in order to provide a clear view of the wider area with a focus on automatic AM in social media text. We present different models that take place in previous research works and break them down to their core tasks. Inspired from the individual AM sub-tasks, we propose a complete conceptual architecture for AM that can be easily adopted and modified depending on the goals of every research work and we present the existing tools in both the wider NLP area and more specified tools for the task of AM.

Argumentation Mining could stimulate a series of applications, where the evaluation and the classification of reasoning is essential, especially in web content, where both reliability and reasoning validity of a user holding a position are questionable. Tasks such as troll detection, knowledge retrieval, and information validation could be significantly benefited by the progress in AM. Cutting- edge techniques in human-computer interaction, opinion mining, and recommendation systems could adopt parts of an AM system and enhance their performance. The successful interpretation, evaluation, and taxonomy of arguments will eventually lead to human level reasoning machines, which can understand, evaluate, and eventually create knowledge.