Socratrees: Exploring the Design of Argument Technology for Layman Users

TL;DR

Socratrees is a user-friendly online platform designed to facilitate structured, collaborative argumentation for laypeople, aiming to improve online discussions and combat misinformation through innovative design balancing structure and ease of use.

Contribution

This paper introduces Socratrees, a novel argumentation platform that combines informal logic principles with user-friendly features for layman users, filling a gap between academic and non-academic systems.

Findings

Most users learned to contribute effectively with minimal guidance.

Users perceived the structured format as beneficial for clarity.

The system may enhance online discussion quality.

Abstract

Terms like 'misinformation', 'fake news', and 'echo chambers' permeate current discussions on the state of the Internet. We believe a lack of technological support to evaluate, contest, and reason about information online---as opposed to merely disseminating it---lies at the root of these problems. Several argument technologies support such functionality, but have seen limited use outside of niche communities. Most research systems overemphasize argument analysis and structure, standing in stark contrast with the informal dialectical nature of everyday argumentation. Conversely, non-academic systems overlook important implications for design which can be derived from theory. In this paper, we present the design of a system aiming to strike a balance between structured argumentation and ease of use. Socratrees is a website for collaborative argumentative discussion targeting layman users, but includes sophisticated community guidelines and novel features inspired by informal logic. During an exploratory study, we evaluate the usefulness of our imposed structure on argumentation and investigate how users perceive it. Contributing to arguments remains a complex task, but most users learned to do so effectively with minimal guidance and all recognized that the structure of Socratrees may improve online discussion and results in a clearer overview of arguments.

Full text

Argumentation,

computer-supported argumentation,

argument technologies,

computational argumentation,

1 Introduction

There currently is an unprecedented amount of information available online. This can be used by decision-makers, the civically-engaged, journalists, and researchers alike, to inform themselves. However, doing so requires navigating and parsing a complicated web of disparate resources (such as news reports, scientific articles, and social media), which can make it hard to ‘see the forest for the trees’. Although we have made great advances in how information can be disseminated, there is a lack of technological support to integrate, evaluate, contest, and reason about it [1].

Argumentation online is certainly possible—and commonplace, but typically only adds to the ever-growing torrent of information, leaving behind lengthy disorganized threads, interspersed with random Internet banter. Since anyone can contribute, verifying the validity of statements found online is extremely time-consuming—time most people do not have or are simply unwilling to give up leisure time for [2]. Instead, most people decide on a select few news sources (e.g., based on political predispositions [3] or attitudes towards specific issues of interest [4]) in which to place their trust and based on which their opinions are formed. As a result, many information sources act like ‘echo chambers’: discussions of opposing views are inconveniently segregated, and ill-founded ideas can propagate freely as they remain largely unopposed [5, 3]. It is no surprise then, that terms like ‘misinformation’, ‘fake news’, and ‘filter bubbles’ permeate current discussions on the state of the Internet [6].

Inspired by argumentation theory [7, 8], research on the border of logic, philosophy, and computer science has taken up the challenge to create better tools to disseminate, structure, and analyze rational thought, collectively called argument technologies. These target a wide variety of application areas, including conflict resolution, legal argumentation in law, discussing and documenting design rationale, and sensemaking [9, 10, 11]. In this paper, we focus on collaborative, publicly available, web-based technologies with explicit support for interlocutors to view and contribute to argumentative discourse. While several such technologies exist [11], they remain a niche outside of research; support for argumentation on mainstream social networks is limited to basic functionality such as up or down voting and conversation threading. Some researchers argue that the primary obstacle to more widespread adoption is a mismatch between the informal nature of online discussions and the highly formal structured functionality current argument technologies provide, i.e., usability [12, 11].

We introduce the design of a website for collaborative argumentative discussion, named Socratrees, exploring the delicate balance between structure—an integral part of argumentation—and ease of use. Prior argument technologies typically rely on complex ontologies to represent claims and relationships between them, e.g., ‘issue’, ‘position’, ‘challenge’, ‘justification’, and ‘agreement’. In contrast, the user interface introduced in this paper reduces argumentation to but three core concepts for the user to understand: (1) statements, that can (2) support and (3) oppose one another. Our primary goal is to allow users to form their own opinions by collaboratively aggregating all information relevant to a given statement in one location. As opposed to prior work (e.g., [13, 14]), our focus is less on analyzing a single argument in great detail and more on providing the necessary structure to represent many competing arguments side by side, without forming judgment as to which one is sound.

Over the course of six weeks, we publicly invited users to try out Socratrees and engaged 18 users in argumentation by responding to the 128 statements they posted on Socratrees. A concluding survey was filled out by 14 users. Based on this exploratory study, we discovered unforeseen challenges and important insights which may be of interest to other designers of argument technologies.

2 Argumentation Theory

Formal deductive logic falls short when trying to evaluate the quality of arguments expressed in ordinary everyday language (such as political discourse) [7, 8]. Practitioners and teachers of logic started challenging the traditional ideals of validity and soundness and it was soon recognized that “[formal logic] had in mind one important subset of arguments, but the realm of argumentation was much broader” [15]. Essentially, the logic of argumentation must be distinguished from formal logic which concerns itself solely with inference/implication; instead, argumentation must be seen as dialectical—a process with arguments as a product of which both sides need to be investigated, for and against, to see how they interact [15].

Today, argumentation theory is an umbrella term for studying the “production, analysis and evaluation of argumentation” by adopting both descriptive and normative methods, i.e., by evaluating argumentative discourse empirically, as it occurs, as well as reflecting on the necessary criteria for reasonable argumentation [7]. Given our focus on providing technological support for argumentative discourse, we are particularly interested in a normative approach since it can prescribe the necessary functionality to enable more effective argumentation. Therefore, informal logic—defined as “the normative study of argument” [16]—became a logical choice as the main driving force behind our design. Walton [8] provides a short introduction to informal logic and presents the following minimal definition of an argument:

An argument is a set of statements (propositions), made up of three parts, a conclusion, a set of premises, and an inference from the premises to the conclusion. An argument can be supported by other arguments, or it can be attacked by other arguments, and by raising critical questions about it.

Adequacy of premises and inferences in informal logic is less strict than in formal logic. Rather than validity and soundness, Blair and Johnson [15] argue for: (1) acceptability (start with premises the audience is willing to accept), (2) relevance (premises ought to be relevant to the conclusion), and (3) sufficiency (premises ought to provide sufficient support for the conclusion). Premises can either work together to form a linked argument or contribute to the conclusion independently, whereby they form a convergent argument. Argumentation schemes prescribe commonly used forms of linked arguments in which each premise plays a specific role, e.g., an argument ‘from expert opinion’ or ‘from analogy’.

With these ‘rules’ of informal logic in mind, there are several ways of attacking an argument [8]. First, argumentation schemes have associated critical questions, e.g., of an argument from expert opinion you can ask, “Is the premise consistent with what other experts in the field assert?” Considering the acceptability criteria, you can either question one of the premises or refute an argument by forming a counter-argument which concludes the opposite. It is worth noting that arguments may attack one another. Lastly, you can argue that a premise is not relevant to the given conclusion, or point out logical fallacies in reasoning.

Based on multiple studies in argumentation theory there is a commonly held belief that humans are inherently bad at reasoning and argumentation, highlighting the need to make ‘critical thinking’ and similar topics part of the core curriculum in education [10]. However, more recent work sketches a less bleak picture of the ‘layman’ arguer [5, 12]. When people are sufficiently motivated, i.e., when argumentation occurs in a dialogical context as opposed to a decontextualized and abstract task frequently employed in studies, they can evaluate arguments quite accurately. Furthermore, in contrast to studies that show that people are bad at producing arguments (e.g., by succumbing to confirmation bias), they form good arguments when challenged and evidence is made available to them. In other words, people are better at evaluating others’ arguments than producing their own. Therefore, argumentation is most effective in groups with heterogeneous views [5].

3 Argument Technologies

Part of the roots of argument technologies can be traced back to some of the earliest work in computer science [17]:

… [the] ‘founding fathers’ of today’s interactive computing such as Bush and Engelbart envisaged argument construction and analysis as a key objective for the intellectual technologies they were conceiving.

Engelbart’s NLS [18], introducing many concepts of personal computing, was designed to ‘augment human intellect’ and Bush’s Memex, frequently cited as anticipating the Internet, was introduced ‘as we may think’ [19]. In later research—still predating the World Wide Web, the essence of hypertext was described as “a computer-based medium for thinking and communication” [20], embodied by systems such as TEXTNET [21] and SYNVIEW [22]. However, given the extensive scope of technologies supporting reasoning that followed [10], we limit ourselves to reviewing those that are publicly available, web-based, collaborative, and include explicit support for argumentation. Schneider et al. [11] have conveniently reviewed exactly such systems (37 in total) and furthermore include a broader discussion of argument technologies. We direct the reader to this excellent review for a complete description of related work.

Here, we provide an overview of the most relevant systems related to Socratrees, while omitting others which might overlap in functionality but target different use cases (e.g., ‘IBIS-like’ decision-support systems [23], knowledge maps, and public opinion polling tools like Opinion Space [24] and Polis111Polis: https://pol.is/). Specifically, we focus on systems that target ‘information seeking’ dialogues in which the goal is to find or share arguments related to a common topic of interest [8].

Common functionality in such systems is to break up arguments into smaller pieces (going by various names, such as statements, premises, claims, and ideas) and specifying connections between them which describe their relationship (e.g., ‘supports’, ‘attacks’, or ‘is similar to’). Relations which denote inferences construct the argument. The resulting underlying data structure is a graph of statements which provides an argumentative overview of a specific discussion.

3.1 Representation and Structure

Although arguments essentially form a graph, they can be visualized in a number of different ways; systems have been classified as linear, threaded, container, graph, matrix, or a combination thereof [10]. Matrix views are extremely uncommon and therefore will not be discussed here.

Linear and threaded representations are most in line with those used in traditional social media, e.g., blogs, and sites supporting conversation threading such as Reddit222Reddit: https://reddit.com (2005) respectively. Argument technologies extend on this. For example, Rich Trellis [25] allows a mixture of arbitrary free text with the ability to annotate relations, resulting in a linear but formalized overview of an argument. Rich Trellis was later extended (yet simplified) to Tree Trellis by reducing the possible types of connections to pro and con and by supporting threading. Videolyzer [26] supports threaded discussions by adding and responding to time-anchored annotations in online informational videos (e.g., mark part of the transcript as a claim, express agreement, or indicate quality).

Container- and graph-based representations are the two most popular approaches in argument technologies. They differ primarily in how connections between statements are visualized. Containers group statements with similar connections to a common ‘root’ statement in a demarcated area, whereas graphs show each connection separately (Fig. 1).

An example of a container-based representation is Debatepedia333Debatepedia: http://www.debatepedia.org (2007), a wiki-based site on which pro and con arguments are grouped together per question on a specific topic. Most recently, Kialo444Kialo: https://www.kialo.com (August, 2017) (likely the most popular argument technology at the time of writing) was introduced. Kialo displays all claims in support of/attacking another claim on opposing sides, and frames them in context of an overarching conversation topic. Although discrete connections are common, some systems provide more granularity. ConsiderIt [27] (now a commercial site555Consider.it: https://consider.it (2016)) aggregates comments on political issues by asking users to register their degree of support on a sliding scale and adding pros/cons to motivate their position. As a result, users can filter pros and cons by degree of support.

Container representations provide a clearer, more concise, overview than graphs, at the cost of reducing the types of connections which can be represented. Therefore, argument analysis tools, with their primary focus on formalizing arguments, typically rely on graph representations. For example, Online Visualization of Argument (OVA) [14] loads text (or a web page) side-by-side with a canvas displaying an associated argument map (Fig. 2). Arguments are mapped by highlighting statements and drawing connections in between them on the canvas. Similar systems are Argunet [13], a desktop tool that supports sharing argument maps online, and the open-source project Arguman666Arguman: http://arguman.org (2014). Common functionality is to allow specifying connections in more detail. For example, OVA supports common argumentation schemes such as ‘practical reasoning’ and ‘expert opinion’, relied upon in Fig. 2. A distinction can thus be made between argument technologies that primarily focus on argument analysis (favoring graph views) and those that focus on information seeking (favoring container views).

In addition to the classification by Scheuer et al. [10], we identify an additional important attribute of argument representation: whether or not a view can be traversed recursively, i.e., whether a statement can be made the center of attention by selecting it, thereby hiding all statements that are not directly connected to the selected statement and showing new ones that are. In contrast to displaying a complete argument map (which can typically be panned and zoomed), only a subset of statements are shown at any given time. Such views are necessary when considering large discussions which may contain cyclical connections and statements that are reused in multiple arguments—a full graph would soon become unintelligible. Most prior work represents single arguments with limited or no reuse of statements, thereby making recursive views surprisingly uncommon. However, Debategraph777Debategraph: https://debategraph.org (2008) [28] is a graph-based system that supports traversing arguments recursively and Kialo supports decomposing arguments into nested hierarchies of pros and cons. Arguments can be explored in great detail by ‘drilling down’ into underlying claims.

3.2 Social Interaction

Most argument analysis tools provide limited support for social interaction, other than editing arguments collaboratively. Conversely, systems with a reduced focus on analysis generally incorporate additional support for collaboration. For example, Cohere [29], REASON [30], and TruthMapping888TruthMapping: http://www.truthmapping.com (2006) target a more general audience and allow users to express agreement with statements. Similarly, Kialo lets users assess the impact of claims on a 5-point scale. Such feedback is incorporated to indicate the strength of statements. Similar to modern social media, argument technologies can trigger notifications when new content is added to arguments a user has contributed to, and content can be moderated (e.g., statements may be reported as spam by users). Kialo seems the most advanced argument technology in regards to supporting social interactions.

3.3 Infrastructure and Integration

Several ontologies have been introduced to define the structure of arguments [9, 11], of which the most promising, still in active development, is the Argument Interchange Format (AIF) [31]. AIF enables sharing of arguments across different online services for argumentation, collectively called the ‘Argument Web’ [32]. For example, AIFdb999AIFdb: http://www.aifdb.org (2012) [33] is a public database for arguments, to which the ‘smoking dialog’ represented in Fig. 2 was uploaded. ArguBlogging101010ArguBlogging: http://argublogging.com (2014) [34] is a browser plugin through which agreement or disagreement can be expressed anywhere online by highlighting text and posting a response to your personal blog and AIFdb. Similar systems (unrelated to the Argument Web) are rbutr111111rbutr: http://rbutr.com (2012) and hypothes.is121212hypothes.is: https://web.hypothes.is (2013), providing a comparable infrastructure to support ‘open annotation’ anywhere on the web. Ontologies are key to automating argument mining, integration, and evaluation [9, 31] (e.g., through the use of artificial intelligence), all of which are considered outside of scope in this paper.

4 Design Principles/Goals

Before presenting Socratrees, we will introduce key design principles which have influenced our design and contrast them with prior work.

4.1 Transparency First—Inspire Critical Thinking

Technology should augment rather than replace human judgment [2]. In line with this recommendation, our goal is not to prescribe what is true or false (i.e., to be a fact finding tool), but to provide transparency to arguments and to inspire critical thinking. Rather than presenting single arguments in great detail, we aim to provide an easily digestible overview of all relevant information in relation to a specific statement. Competing arguments thus live side-by-side and it is up to users to interpret them and draw their own conclusions. In other words, our main goal is not supporting ‘argumentation’ per se, but providing a record of the collaborative thought process in order to aid individual human reasoning. Such overviews facilitate distributed sensemaking [35], and group reasoning in which argumentation theory predicts truth to win out [5].

4.2 Help Finding Common Ground

Argumentation theory describes different types of dialog with differing goals: e.g., in a persuasion dialog the goal is to convince the other party, and in a deliberation dialog the goal is to decide on the best available course of action [8]. Argument analysis systems (as discussed in related work) primarily target persuasion dialogs. In contrast, Socratrees targets information-seeking and inquiry dialogs, in which the goal is to exchange information and find and verify evidence respectively. By sharing statements in relation to one another (as supporting or opposing) and allowing users to express agreement with each, an overview becomes available of how well-supported statements are, the different reasons for believing or not believing in them, and how popular these are. Strong arguments on either side of a discussion indicate ‘common ground’, whereas statements on which opinions are divided reveal true points of contention.

4.3 Conducive to Large-scale Discussions

Information-seeking and inquiry dialogs are cooperative in nature, as opposed to persuasion dialogs which tend to be adversarial, as summarized by the philosopher Neurath [36]:

Debaters on comprehensive scientific problems are … like lawyers who have to take a side. Each of them intends to strengthen his own arguments and to weaken the arguments of the aggressor—but no judge is in the chair. … Finally we find ourselves all together in the same ship and are co-operating even when we think we are fighting one another.

To better support the cooperative nature of argumentation, we explore interaction techniques for large groups of users to collaboratively contribute to discussions comprising numerous statements, without hindering one another, and while retaining a suitable overview for people arriving later to, or having missed part of, the discussion.

Based on our review of web-based argument technologies, we identify three core challenges to supporting large-scale discussions online: (1) dealing with digressions in order to ensure focused argumentation, (2) supporting users to explore arguments at their own pace—without being overwhelmed by expert accounts, and (3) enabling/encouraging statement reuse in order to eliminate redundant discussions and to capitalize on prior knowledge. TruthMapping has a built-in mechanism to deal with digressions, and Kialo supports focused exploration of arguments by breaking them up into concise claims. However, statement reuse in prior systems is rare or simply unsupported. Kialo supports reusing claims through the use of ‘symlinks’, but in practice this feature is hardly used. We observe that for statements to be reused, it is essential that they are ‘free of context’, i.e., that they can be interpreted outside of the context in which they were introduced. Prior systems do not enforce this.

4.4 Inclusiveness

Prior academic work has mostly enforced argument structure, requiring knowledge of argument analysis, at the cost of usability [12, 11]. We believe that given a more suitable medium, anyone can contribute meaningfully to argumentation online [5, 12]. Users should not be expected to know argumentation theory in order to start using argument technologies. Similar to Cohere [29] and the non-academic systems TruthMapping and Kialo, we target a more general audience. Our goal is to find a suitable balance between imposing structure and ease of use. In addition, we aim to be non-discriminatory; it should always be possible for minorities and repressed groups to share unpopular or controversial beliefs.

5 Socratrees

Socratrees131313Socratrees: http://socratrees.wiki/ (Fig. 3) is a collaborative, web-based, argument technology, targeting a general, non-expert, audience. The main user interface supports outlining arguments into hierarchies of supporting and opposing statements and relies on a container representation which can be traversed recursively to navigate between statements (as described in Section 3.1). This supports focused argumentation and helps dealing with digressions. Kialo is the only other system combining a container representation with recursive navigation. In addition, Socratrees introduces two novel features to structure arguments: (1) statements can be represented both in a normal and negated form, and (2) statement relations (one statement supporting/opposing another) are considered statements themselves which further supporting/opposing statements can be added to.

First, we present the structure of Socratrees and our motivation behind it in more detail, including how navigation is supported. Second, we explain the chosen visual metaphor and how we intend to evolve it over time. Lastly, we present community features and how we believe they can be used to inspire and scale up argumentation.

5.1 Structure and Navigation

We have chosen to only support two types of relations between statements—supporting and opposing—similar to earlier argument technologies targeting a layman audience. We find this to be a suitable balance between structure and ease of use. Additional semantics are implicit and we trust users to be able to infer more concrete relations based on context. For example: “Pineapple does not belong on pizza” is supported by “Italians don’t put pineapple on pizza” and opposed by “Hawaiian pizza is very popular in Australia”. An explicit analysis of these arguments is unnecessary to understand the two points made. We have opted to label relations as ‘supporting/opposing’, in contrast to the more common ‘pro/contra’ and ‘supporting/attacking’. Early feedback indicated this alternative wording inspired animosity and one-sided thinking, counter to our design principles (‘finding common ground’). For the same reason, supporting and opposing statements are depicted using the same color, in contrast to prior work which mostly relies on green and red respectively. Although differing colors make sense in light of the ‘redundancy gain’ principle in Human-Computer Interaction (HCI), we find it more important to highlight what statements have in common rather than what sets them apart; all relevant statements contribute to the discussion in a meaningful way.

This similarity is further emphasized by the following observation: when considering the negated form (inverse) of a statement, all supporting statements become opposing statements, and vice versa. For example: “Governments should defend freedom of choice of its citizens” (as highlighted in Fig. 3.H) becomes a supporting statement when considering that “Governments should not ban smoking” instead of “Governments should ban smoking”. Socratrees exploits this fact by supporting both a normal and negated form for statements. By clicking the ‘inverse’ icon (Fig. 3.A), a negated form of the statement is shown and the supporting and opposing statements are swapped. A default “(not)” text is prepended to the normal form of statements, but users can specify a custom, more suitable, text for the negated form. This has two major advantages which reinforce our underlying design principles: (1) better statement reuse since the normal and inverted form of statements are structurally identical, and (2) discussions on unpopular or controversial ideas (e.g., holocaust denial) are directly linked to the arguments against. This makes suppressing them nonsensical as it would also eliminate the arguments in support of what the majority of society believes to be factual or ‘true’; this supports our ‘inclusiveness’ design principle. To our knowledge, Socratrees is the first system supporting both a normal and negated form for statements.

The last mechanism available in Socratrees to structure arguments is related to the requirement of relevance. Arguing whether a statement is relevant is an essential, common, part of argumentation, supported in traditional argument analysis tools by applying argumentation schemes and warrants to statement relations. Systems targeting a more general audience like Kialo, Consider.it, and TruthMapping do not provide support for this, presumably to sidestep this additional complexity in favor of ease of use. We try to find middle ground by repurposing the basic building block of argumentation—the statement—to represent statement relations as well. Concretely, when adding a statement $A$ as a supporting/opposing statement to statement $B$, the system introduces this relation as the statement “$A$ supports/opposes $B$” with the inverse form “$A$ does not support/oppose $B$”. Similar to how underlying statements can be accessed recursively, the relation statement can be accessed by clicking the stalk which connects the statement to the argument ‘tree’ and further underlying statements can be added to it. For example, following the relation of the statement highlighted in Fig. 3 leads to the page shown in Fig. 4. The newly created statement thus acts as the starting point for a discussion on whether or not the given statement is relevant in relation to the other (“Does A support B?”), reusing concepts the user is already familiar with. It can also be used to construct linked arguments (statements working together to reach a conclusion). Further stalks within the statement relation are hidden since reasoning about them becomes overly complicated (“Does C support A supporting B?”) and the need to discuss them has not arisen during early testing.

For these structural features to work as intended, we introduce three requirements statements need to comply with:

Be concise: Keeping statements concise keeps discussions focused, makes statements more likely to be reused, and makes interpreting their negations easier. When possible, conjuncts should be avoided by splitting statements into two. As argued in the design of Tree Trellis [25], this comes at the expense of expressiveness, but also improves the ability to elaborate on arguments without restructuring them, and ensures ancillary points are not mixed with more central ones. To enforce this, statements have a maximum of 120 characters; a limit which did not inhibit us from expressing our own arguments during early testing.

Aim to be free of context: It should be possible to interpret a statement outside of the context within which it was introduced. For example: “Wind turbines are more effective” makes sense in relation to a statement about solar panels, but takes on a different meaning when used in the context of nuclear energy. To ensure effective reuse of statements, statements should thus be able to stand on their own and eliminate indexical references (e.g., ‘I’, ‘also’, ‘this’, and ‘that’).

Not be phrased as questions: Questions do not contribute concrete information to the structure of arguments, and rather indicate specific information is lacking or missing. In essence, the answers to questions are what constitute valid statements on Socratrees. This last requirement is controversial due to the unmistaken value of critical questions in discussions. However, critical questions can still be posted as comments on statements (Fig. 3.B).

Navigating arguments is supported by traversing statements recursively. At any given time, only a single statement with its supporting and opposing statements is displayed. Opening a different statement is done by clicking on it. While this ensures focus and allows users to explore parts of an argument most relevant to them (in line with our design principles), this becomes disorienting when following many links. To see which statements you followed to reach your current position in a tree, a ‘current branch’ section is displayed in the sidebar (Fig. 3.C). Previously visited statements can be revisited by clicking on them, without erasing the stored navigation path. E.g., Fig. 3 shows the path followed to reach the previously visited statement “Nicotine is an addictive drug”. Only when diverging from the presented path, the stored navigation path is overridden. The sidebar also shows which other statements the current statement is used in (Fig. 3.D).

New statements can be added after having searched for existing statements first (Fig. 3.E). An intermediate screen shows search results and an option to proceed with adding the statement in case the desired one is not found. Adding supporting and opposing statements works in a similar way (Fig. 3.F). Importantly, the search results page also allows users to view and select the normal or negated form of statements, since the form to choose depends on the argument you want to make. Enforcing ‘search first’ further encourages statement reuse. Statements are anonymous in order not to discourage people from expressing unpopular beliefs. However, comments are linked to usernames.

5.2 Visual Metaphor

After considering other visual metaphors (‘branching rivers’ and ‘neural networks’), we eventually chose to represent statements and their supporting and opposing arguments as leaves on a tree. This is a visually rich metaphor that everyone can relate to and evokes a sense of tranquility which might counteract the adverse nature of argumentation. When you agree with or believe in a statement you can ‘give it water’, represented as droplets (Fig. 3.G). Although our current exploitation of this metaphor is limited, we chose to represent statements as leaves since the physical properties of a leaf correlate nicely with those we need to represent. Statement relevance (the statement relation) is presented by the stalk leading up to the leaf. The stalk might be ‘broken’ when a statement is deemed irrelevant as determined by underlying statements. Leaves in turn can have different colors depending on their ‘health’. Once we have explored the properties of what makes a good or bad statement in more detail during an extended evaluation we intend to include such visualizations.

5.3 Community Features

Although our primary focus in this first iteration of Socratrees was to explore an alternative structure for argumentative discourse, we implemented basic communication, notification, statistics, and moderation features which provide a strong basis for future work.

We recognize that the potential of Socratrees lies not in replacing existing discussion platforms, but augmenting them. Therefore—by design—we provide limited support for unstructured argumentation; free form comments can be added to each statement (Fig. 3.B) to add unstructured thoughts, but their primary purpose is to question, clarify, and add related resources. Lengthy discussions are discouraged141414This is inspired by comments on Stack Exchange (https://stackexchange.com/) which are seen as transient. Valuable comments should be incorporated in the main content of the site (questions and answers).. We count on external websites, less restrictive in form and embodying richer communities, to link to content created on Socratrees and extend on discussions there. To enable this, each statement has a representative URL (e.g., “/statement/657/governments-should-ban-smoking”), and users can link directly to individual comments and statement relations which causes the linked content to be highlighted (Fig. 3.H). Furthermore, for our exploratory study we integrated with a Reddit discussion board. Clicking a Reddit icon linked to the creation of a new Reddit post with a matching title and a link back to the specific content on Socratrees (Fig. 3.I).

The first few statements new users post are posted as drafts (Fig. 3.J). Draft statements are under review and cannot be added to or used as related statements until they are approved. We implemented this feature so that moderators can provide feedback in comments to new users in case their statements do not comply with site guidelines. Moderators can also turn problematic statements into drafts. For now, moderators are predetermined, but we intend to adopt a reputation-based distributed moderation model similar to Stack Exchange [37].

Users can add ‘droplets’ to leafs and stalks (statement relations) they believe in. The accuracy of statements can thus be judged independently of their relevance. The total number of users agreeing with a statement or its relevance is displayed next to the droplet. In addition, a leaf icon indicates the total number of underlying statements (Fig. 3.K). To express belief in the negated form of a statement the inverse view needs to be loaded first. This forces users to view underlying statements prior to expressing their disbelief—the equivalent of ‘down voting’ in traditional social media. We hope this may inspire users to clarify their disagreement, which we deem more constructive than mere down voting. The droplet icon is grayed out when you have not formed any opinion and shows a ‘minus’ when you have expressed belief in the inverse form of the currently shown statement (Fig. 3.L). The result is an overview of all the statements you have formed an opinion on.

Adding droplets also acts as a subscribe mechanism; users receive notifications about changes to any of the statements they have expressed belief or disbelief in. We feel expressing your opinion should go hand in hand with a willingness to participate in a discussion about it, and have therefore opted to combine these two features into one. Lastly, a top bar shows an inbox for notifications, in addition to a link to the user’s account, the number of times people have expressed agreement with statements added by the user (regardless of form), and the number of approved statements posted by the user (Fig. 3.M).

6 Exploratory Study

Early mock-ups based on the analysis of Reddit threads proved to be too limiting to evaluate the feasibility of our proposed structure for argumentation. To explore how statement negations and our representation of statement relevance influence structuring and interacting with arguments, a functional system had to be built—these features are missing in prior work. In this sense, the presented system can best be described as an ‘interactive sketch’ [38]:

… early designs can also be implemented and maintain their sketch-like properties, … to explore the ideas behind highly interactive systems. When systems are created as interactive sketches, they serve as a vehicle that helps a designer make vague ideas concrete, reflect on possible problems and uses, discover alternate new ideas and refine current ones.

Thus, the main purpose for implementing Socratrees was to “evaluate the provided functionality to structure arguments” as specified in the preregistration for this study151515Preregistering studies is recommended by Cockburn et al. [39]. The preregistration for this study is available on AsPredicted: https://aspredicted.org/blind.php?x=7p26xn. This evaluation is a first step towards answering the overarching research question: “how to strike the right balance between introducing functionality for structured argumentation and usability in order to obtain more widespread adoption of argument technologies?”

The author of this paper analyzed three arguments in detail and added them as statements on Socratrees: a popular blog post on a controversial topic in software development (having received over 100,000 views and 100 comments), a discussion on whether or not Socratrees improves online discussions, and a smaller argument about smoking taken from literature on informal logic [8], part of which is represented in Fig. 3 and Fig. 4. These and other smaller arguments were added successfully, based on which we conclude Socratrees is capable of representing a diverse set of real-world arguments.

Next, we evaluated Socratrees over the course of six weeks involving 32 users. We wanted to assess to which degree users agree our proposed structure has the potential to achieve the design goals outlined earlier: (4.1) transparency first—inspire critical thinking, (4.2) help finding common ground, (4.3) conducive to large-scale discussions, and (4.4) inclusiveness. The website was announced to friends and colleagues in person, on social media, and several public websites (e.g., subreddits dedicated to argumentation and critical thinking). 238 unique users visited the site, out of which 59 signed up to participate in private beta and were granted access. Given the lack of extensive moderation features, we restricted public access so that only users with access could view statements and contribute to the site. 27 users did not visit the site after having received access, based on which we conclude that 32 users in total participated (to some degree) in private beta. This includes the author of this paper who acted as a moderator (approving draft statements and commenting on them), in addition to acting as an ever-present ‘active’ community member who engaged new users in discussions. Without moderation, users would be able to add statements that do not comply with site guidelines, which as discussed earlier are essential for our structural features to work as intended. Lacking a larger community that has a thorough understanding of site guidelines, we believe this was the only reasonable way to mimic an active community and what we eventually hope to enforce by applying a distributed moderation model similar to Stack Exchange [37].

19 users added 374 statements and 371 statement relations, so 13 users only browsed the site. Because the evaluation required participation by the author, the majority of statements were added by him, including the three arguments which were analyzed in detail: 246 statements and 286 relations. Thus, the remaining 18 users added 128 statements and 85 relations. Fig. 5 shows an overview of all connections between statements, laid out so that individual arguments and connections between them can be recognized. This gives an impression of argument complexity and why it was necessary to evaluate this structure using an interactive system. 45 statements (12%) were used more than once (as supporting/opposing). Negated forms of statements contributed to statement reuse and lead to a particularly interesting use case: at times, both forms of a statement were added to opposing sides of an argument, thereby highlighting key points of disagreement (e.g., “Global warming is/is not man-made” supports/opposes “Climate change is man-made” in Fig. 5.A). The average text of approved statements was 58 characters long ($N=367$, $SD=24.0$, $Min=16$, $Max=119$).

All active users received an online survey at the end of the study, which was filled out by 14. Our survey data shows a wide variety of users: the median age was 28.5 ($SD=11.3$, $Min=23$, $Max=64$), 8 male and 6 female, with backgrounds such as data science, HR consulting, IT, and project management. On a 5-point Likert scale from “Not at all familiar” to “Extremely familiar”, nobody specified familiarity with argumentation theory higher than “Neutral” ($\mu=2.1$, meaning “Not familiar”). The median number of hours users reported having used the site was 2 ($SD=2.8$, $Min=0.3$, $Max=10$).

Five central survey items made claims about the potential of Socratrees and had to be rated on a 5-point Likert scale from “Strongly agree” to “Strongly disagree”. Each scale provided the option to enter free form feedback to clarify the provided rating. Users were instructed as follows: “We are interested in your assessment of how well the design of Socratrees might support argumentation online. To this end, we are primarily interested in feedback related to the core structure represented in the user interface.” To make sure users understood what was implied by ‘core structure’, a short summary of structural features was included. Each question started with “Socratrees …”

Q1–design goal 1 “… can help understanding arguments”

Q2–design goal 2 “… can help finding common ground”

Q3–design goal 4 “… is non-discriminatory, i.e., can represent the opinion of minorities and repressed groups”

Q4–design goal 3 “… can host conducive, noninflammatory, discussions”

Q5–design goal 1 “… inspires critical thinking”

The distribution of responses (Fig. 6) indicates most users believe Socratrees has the potential to achieve the goals we originally laid out in our design principles. Free form feedback was aggregated using thematic analysis and we will present a representative quote for each individual finding next.

(Q1) 11 users elaborated that the division of supporting/opposing statements provides a “convenient” and “quick overview” which “makes it easier to understand arguments” and “figure out why two opinions differ.” However, one user noted that “the writer’s definition of a term [might be] inconsistent with the reader’s”, potentially leading to conflicts in understanding.

(Q2) Most users indicated that “[b]y separating arguments into small and separate discussions, it becomes easier to see where you agree/disagree” and you “are more likely to read both sides”. They noted that you are more likely to agree on a single statement than on a whole argument; “people can agree on having different value-sets.”

(Q3) A common concern was Socratrees requires a “special skill set only common in people with higher educations”, e.g., having a “sense of how important details can be in a [statement]”. And, while it “allows for expressing [anonymous] opinions, regardless of their popularity”, two users expressed concerns about “counterfactual” statements. Allowing widespread, uncontrolled, discriminating statements “would run counter to the goal of being non-discriminatory.”

(Q4) Four users shared the belief that “[t]he structure of the platform leaves little room for personal attacks and puts the topic at hand at the core of any discussion.” Three others argued that discussions are inherently “emotional” and will always elicit “strong reactions”. Socratrees can “temper [this] but can’t eliminate it”.

(Q5) Users overwhelmingly agreed that “[s]tructured argumentation encourages … users to understand the relationship between statements.” Socratrees “forces” users to “think before [they] write”, make the implicit explicit, and “to be very precise in the relation between [statements]”. However, you “have to spend a tremendous amount of energy to molding your statements to this structure and all its rules” and some people “will never have the interest in exploring views that are different to their own”.

Lastly, we asked users what would encourage them to contribute more to a site like Socratrees. Their primary request was a “larger community” with “topics that are relevant to current issues.” Users want to see that “[their] statements are making a difference” and to this end envisioned features such as sharing statements from external websites (supporting “evidence-based links”), enhanced reward mechanisms (gamification), and identifying like-minded individuals based on tags for “ideologies and beliefs”. The lack of such features left some users wanting for a “purpose” and “clear goal”.

7 Discussion—Challenges

The results of our study indicate that the structure for argumentation introduced in Socratrees shows much promise. Even though argumentation remains a complex task, compared to how it currently takes place on social media, users learned to appreciate and understood the trade-offs made, as indicated by our survey results. When asked what would encourage users to contribute more, they did not request a simplified design. Rather, they envisioned scenarios in which a broader community would adopt the given technology. However, the design was not without flaws. The author of this paper used Socratrees extensively to structure his own arguments and moderated all content added to the site, based on which a thorough understanding of limitations to the current design arose. In addition, multiple users provided feedback via private communication channels and on the Reddit discussion board, which also contributed to a thorough understanding of open challenges.

Transitioning from Unstructured to Structured Content

Comments were underused and not fully understood by users. Even though they were designed as a staging ground for thoughts that could not yet be phrased as concrete statements, some users—unaware of this feature—initially expressed a desire to add less-structured argumentation. Comments thus need more exposure and additional mechanisms need to be put in place to go from unstructured to fully-structured argumentation, e.g., incremental and system-assisted formalization mechanisms [40].

Reasoning about Relevance is Hard

Adding statements to statement relations in order to discuss relevance is confusing and particularly error-prone (i.e., adding statements to the statement relation depicted in Fig. 4). While we maintain this is an essential feature, it requires keeping track of three statements and two statement relations in parallel, which can be cognitively demanding. Furthermore, it is not always clear whether two statements should be added as one arguing for the relevance of the other, vice versa, or both. For example, “All questions seek knowledge” supports “There are no stupid questions”, which is relevant because “Seeking knowledge cannot be stupid”. But, an alternate phrasing would be “Seeking knowledge cannot be stupid”, which is relevant because “All questions seek knowledge”. We see no immediate harm in statements ‘reinforcing’ one another like this and adding both forms to the argument graph, but these findings indicate the need for usability testing and trying out alternate visualizations.

Differing Specificity of Similar Statements

Very similar statements require entirely different arguments. Yet, by adding arguments, some users were implicitly changing the specificity of the statement they were contributing to. For example, statements regarding second-hand smoke were being raised in a discussion on whether “Governments should ban smoking”. This is more relevant in a discussion on whether “Governments should ban smoking in public spaces”. Such related yet disparate statements need better support to keep discussions more focused.

Comparative Statements

Many deliberation dialogues—as opposed to information seeking and inquiry dialogs we design for—are comparative in nature, for which the platform is not perfectly suited. For example, the main statement analyzed in the software development blog post was: “Underscore formatting is more suitable than camel case formatting for code”. For the vast majority of underlying supporting/opposing statements, this requires repeating this comparison to ensure statements are ‘free of context’. For example: “Underscore formatting of code is faster to read than camel case formatting”. On the other hand, these types of discussions are potentially flawed as they often reflect ‘false dichotomies’; they might rule out other or intermediate options. Emphasizing pairwise comparisons when an argument relies on them has the potential benefit of highlighting the limited scope of statements.

Dependence of Statements on Context

More generally, writing statements that are ‘free of context’ is unintuitive and hard. Many of the first statements added by new users were not living up to this guideline. Suchman argues that “the communicative significance of a linguistic expression is always dependent on the circumstances of its use” [41], making it impossible to remove all dependencies on external context. However, in practice it is possible to elaborate on meaning to such a degree that the likelihood of misunderstandings arising due to differences in interpretation becomes minimal. The problem is this requires excessive reiteration of context in each supporting and opposing statement, which becomes longer and longer as statements relate to more specific situations. To resolve this, we envision ‘context tags’ which can be added to statements to introduce implicit context with an associated definition. When adding a related statement they are inherited by default, but can be removed by users in case they feel a statement can be made more generic.

8 Conclusion

Overall, we conclude that the structure imposed on argumentation by Socratrees is a great first step towards accomplishing our goals to make arguments transparent, inspire critical thinking, help find common ground, and be conducive to discussion—at the cost of making it harder to contribute to arguments. We identified challenges for future research and reported on insights which may be of interest to other designers of argument technologies. However, the general observation that structuring one’s thoughts in ‘network structures’ is challenging is nothing new [17]:

… argument mapping initially feels like learning a new foreign language, and the temptation is to lapse back into more familiar languages (conversational patterns and modes of writing). The tools can be made user friendly, and the notations lightweight and informal, but the human element of the system must co-evolve as well.

While users should not have to know argumentation theory (none of our respondents did) in order to participate in argumentation, it would be overly optimistic to hope to eliminate the ‘hard work’ that goes into formulating an effective argument. The very act of identifying statements and relationships between them is what defines critical thinking. Therefore, we can only hope to inspire users to engage in and learn this alternate form of literacy by providing better support and making the work more rewarding and fun.

However, this does not mean we intend to replace traditional writing or believe that all arguments should be constructed as argument maps. Socratrees is designed to augment linear text. Our long-term goal is to be able to highlight statements anywhere on the web (relying on an ‘open annotation’ infrastructure similar to hypothes.is and rbutr) and linking them to structured argumentation. We feel this could effectively replace traditional comment sections on news articles, blog posts, or even scientific papers, and provide better support for people to critically evaluate, contest, and reason about information online. At a glance, the popularity of statements and how well-supported or controversial they are could be assessed, and more advanced statistics might be able to indicate how objective or biased an article is. Looking even further ahead, making arguments more focused and open to falsification has the potential of scaling up governance and by extension the democratic process [1].