Quantifying the Vulnerabilities of the Online Public Square to Adversarial Manipulation Tactics

TL;DR

This paper models social media information diffusion to quantify how adversarial tactics like infiltration and flooding can degrade content quality, revealing vulnerabilities and potential countermeasures to enhance platform resilience.

Contribution

It introduces a simulation-based approach to evaluate the impact of manipulation tactics on social media content quality, highlighting the role of influential accounts and targeted strategies.

Findings

Infiltration of communities is most effective in spreading low-quality content.

Influential accounts amplify community vulnerabilities to manipulation.

Targeted attacks on specific users can mitigate overall harm.

Abstract

Social media, seen by some as the modern public square, is vulnerable to manipulation. By controlling inauthentic accounts impersonating humans, malicious actors can amplify disinformation within target communities. The consequences of such operations are difficult to evaluate due to the challenges posed by collecting data and carrying out ethical experiments that would influence online communities. Here we use a social media model that simulates information diffusion in an empirical network to quantify the impacts of several adversarial manipulation tactics on the quality of content. We find that the presence of influential accounts, a hallmark of social media, exacerbates the vulnerabilities of online communities to manipulation. Among the explored tactics that bad actors can employ, infiltrating a community is the most likely to make low-quality content go viral. Such harm can be further compounded by inauthentic agents flooding the network with low-quality, yet appealing content, but is mitigated when bad actors focus on specific targets, such as influential or vulnerable individuals. These insights suggest countermeasures that platforms could employ to increase the resilience of social media users to manipulation.

Equations

$$Q_t=\frac{1}{\sigma N}\sum _{i=1}^N\sum _{m\in M_i}{q}_{i,m,t},$$

Full text

Significance Statement: We show that social media users are vulnerable to adversarial manipulation tactics, through which bad actors can amplify exposure to content that potentially threatens democratic elections and public health. While tactics such as flooding the network with low-quality yet appealing content are damaging, getting users to follow inauthentic accounts has the most detrimental impact. Bad actors can increase harm by maximizing coverage rather than targeting particular individuals, such as influential ones. The varying degrees of harm associated with these tactics highlight tradeoffs and specific areas on which platforms could focus as they develop deterrence strategies against manipulation.

1 Introduction

The vision of social media as the modern public square has been challenged as users have become victims of manipulation by astroturf (1, 2), trolling (3), impersonation (4), and misinformation (5, 6, 7). False news have been reported to spread virally — similarly to reliable information (8) or even more (9) depending on operational definitions. These kinds of manipulation exploit a complex interplay of socio-cognitive (10, 11), ideological (7), and algorithmic (12, 13) biases. The exploitation is enabled or greatly facilitated by inauthentic accounts that impersonate people with malicious intent. Many such accounts can be coordinated by a single entity (14), either manually or through software applications commonly known as social bots (15, 16). Inauthentic and/or coordinated accounts have been observed to amplify disinformation (8), influence public opinion (17, 18, 19, 14), commit financial fraud (20, 14), infiltrate vulnerable communities (21, 3, 22), and disrupt communication (23, 24).

Can social media be manipulated to the point that they no longer function as a public square? Under what conditions? It is difficult to carry out empirical experiments and analyses in the real world to explore these questions (25, 26). One challenge is the limited size of experiments in the wild, stemming from both costs and ethical concerns about the potentially harmful nature of content from bad actors. A second difficulty is the limited data from social media platforms available to researchers (27), exacerbated by recent events such as the acquisition of Twitter by Elon Musk. These difficulties have led, for example, to conflicting accounts about whether disinformation campaigns on social media can sway elections (28, 29, 30, 31, 32, 33). Evidence suggests that these operations mainly impact specific vulnerable communities (34, 35). However, we lack a comprehensive quantitative understanding of how coordinated inauthentic tactics can disrupt online communities. This prevents the informed design of moderation or regulatory policies to protect the online public square from adversarial manipulation.

Here we introduce SimSoM, a minimalistic model of a generic social media platform. The model allows us to explore scenarios in which an information-sharing network is manipulated by malicious actors controlling inauthentic accounts, and to measure the consequences of such information operations. We assume that bad actors aim to spread low-quality information. While there are different kinds of low-quality content in reality — disinformation, conspiracy theories, malware, or other harmful messages — our model uses an abstract definition of low-quality content that encompasses these different types. The impact of the manipulation is measured in terms of the quality of information to which users are exposed in the network.

We find that the presence of manipulation is sufficient to suppress quality information, driving low-quality content to spread virally in the network. We also examine network vulnerabilities that may amplify the effects of manipulation, and evaluate the overall information quality as the result of different malicious tactics, such as infiltrating a community, generating attention-grabbing content, flooding the network, and targeting specific individuals. Insights from these analyses are instrumental in developing countermeasures to increase the resilience of social media and their users against manipulation. We discuss mitigation steps that platforms could take and the issues that arise from regulations aimed at protecting human speech from suppression.

Results

We model information diffusion in a social media platform such as Twitter/X, Instagram, or Mastodon. The information system is a directed network with nodes representing accounts and links representing follower relations. Similar to real-world platforms, content circulates through messages that appear in news feeds. Agents can post new messages or reshare messages from their feeds, generated by their friends, i.e., the accounts they follow. The information diffusion process is illustrated in Fig. 1. Messages represent information that could take the form of text, links, hashtags, images, or other media. An agent can introduce a new message into the system or, alternatively, select a message from their news feed to reshare. Messages created and reshared by an agent then appear on the news feeds of their followers.

Even though people prefer quality content (36), their sharing behavior is mediated by other factors such as laziness (37) and message appeal. To account for this, each message $m$ in the model has two intrinsic and independent attributes. The appeal $a_{m}$ models the likelihood that the message is actually reshared by agents (Fig. 1). The quality $q_{m}$, on the other hand, represents objective, desirable properties of content such as the originality of an idea or the accuracy of a claim. Here we naively represent quality as a scalar value. Deceptive posts may have low quality yet high appeal. For example, false news and junk science articles have low quality — most people would not share them knowingly. Yet such low-quality content may be even more likely to spread virally than high-quality information (9). Low-quality content may be novel, clickbait, ripped from headlines, and/or may appeal to people’s political, emotional, or conspiratorial bias. Worse yet, bad actors can employ generative AI to produce such content at scale (38).

The model captures bias towards appeal as well as two other ingredients that are prioritized by the ranking algorithms of social media platforms, namely social engagement and recency (39). An agent selects a message to reshare from the news feed, which is an inventory of distinct messages recently shared by the agent’s friends. The message is selected with probability proportional to: (i) its appeal; (ii) its social engagement, defined as the number of times it has been shared by the agent’s friends; and (iii) its recency, which decreases with time in the feed (see details in Methods).

Unlike authentic agents, whose intention is to consume and share high-quality information, we define inauthentic agents as accounts that are controlled by bad (adversarial) actors to spread low-quality content among authentic agents. We refer to these accounts as “bad actors” or “inauthentic agents” throughout this paper. Such accounts may be controlled by humans (trolls), software (social bots), or a mixture (cyborgs). The model has three parameters to model manipulation tactics by bad actors: infiltration, deception, and flooding, explained next.

Infiltration describes how bad actors amplify exposure to their messages by getting authentic accounts to follow them (Fig. 2(a)). Bad actor infiltration into the social network is modeled by a parameter $\gamma$, the probability that a bad actor is followed by an authentic agent. Unless otherwise stated, we assume that authentic agents follow bad actors uniformly at random. Fig. 2(b) illustrates the effective suppression of information quality when $\gamma$ is high.

The quality $q$ and appeal $a$ of messages originating from authentic accounts are drawn independently from two distinct distributions, reflecting empirical evidence that these messages tend to have high quality and low appeal (see Supplementary Material). In contrast, we assume that bad actors can manipulate information in the network by creating messages with low quality ($q=0$) and deceptively high appeal. The appeal differential of content from bad actors is modeled by the deception parameter $\phi$, defined as the probability that bad actor content is irresistible ($a=1$). In the absence of deception ($\phi=0$), the appeal of bad actor messages is drawn from the same distribution as those from authentic accounts (see details in Methods).

Flooding is another tactic inauthentic accounts can use to amplify their influence, by crowding out high-quality information. To model this, the parameter $\theta$ is defined as the ratio between the exposure of bad actor content and authentic content (see Methods). Bad actors can achieve high exposure in several ways, including posting at high frequency/volume and artificially inflating popularity/engagement indicators (8, 40, 38).

SimSoM lets us explore information diffusion on social media, including the properties of authentic accounts that might render them vulnerable to adversarial attacks and the effects of different manipulation tactics. In the next sections, we present the results from simulations of the model on online communities derived from an empirical follower network ($N\approx 10^{4}$ Twitter accounts). The network has both scale-free structure (hubs) and high clustering (triangles), structural characteristics that are ubiquitous in socio-technical networks. It also has a realistic community structure, with two well-separated groups of accounts capturing political polarization (see Methods).

Once the system reaches a steady state, in which the message quality across the network has stabilized, we record the mean quality of the messages in the feeds of authentic agents. These measurements are further averaged across simulation runs with the same parameters but different random seeds. We simulate the information diffusion process in networks with different structures to explore whether social network features render communities vulnerable to bad actor tactics. Similarly, we evaluate the impact of these tactics by evaluating the model with varying levels of bad actor infiltration ($\gamma$), deception ($\phi$), flooding ($\theta$), and targeting specific types of accounts. We report on the relative quality, defined as the ratio of the average quality of authentic agents to that of a baseline without bad actors (see Methods).

Network Vulnerabilities

Key structural features of the social network may play a role in amplifying our vulnerability to manipulation by bad actors. The empirical network has two features that are ubiquitous in social media: the presence of hubs and highly clustered communities. We can explore how overall quality is affected by these features through three alternative networks constructed by shuffling links while preserving hubs, community structure, or neither (see Methods). Fig. 3 shows that community structure does not significantly affect the overall quality (purple vs. blue) because we assume that agents in each community are equally likely to follow bad actors. On the other hand, having hubs makes a network more vulnerable to manipulation: the relative system quality is significantly lower in the presence of hubs, both when there are communities (purple vs. green, $p<10^{-4}$) and not (blue vs. orange, $p<10^{-4}$). This is because node in-degrees and out-degrees are highly correlated (Spearman correlation 0.9, $p<10^{-4}$) in the empirical network. Therefore, when authentic followers are concentrated among hubs, high-quality content is also concentrated among those hubs. This implies that high-quality content has more competition and becomes obsolete more quickly compared to the case in which this content is uniformly distributed among authentic nodes. The same does not apply to content from bad actors because this content spreads uniformly among authentic users. (The scenario in which content from bad actors mostly concentrates among hubs is explored later.)

Infiltration, Deception, and Flooding Tactics

Bad actors may maximize their message spread by combining various manipulation tactics. We systematically quantify the effects of these tactics through simulations varying the parameters for infiltration ($10^{-4}\leq\gamma\leq 10^{-1}$), deception ($0\leq\phi\leq 1$), and flooding ($1\leq\theta\leq 32$). See Methods for further details. Fig. 4(a,b,c) illustrates the effects of individual malicious tactics on the overall quality of information spreading through the network, compared to a baseline scenario without bad actors. We observe that infiltration is the most harmful manipulation tactic: when authentic agents have a $\gamma=10\%$ probability of following each bad actor, the average quality in the system is reduced to less than half. Flooding and deception have smaller effects. When low-quality content has $\theta=32$ times more exposure than authentic content, quality is reduced to less than 70%. When bad actors generate content with maximum appeal exclusively ($\phi=1$), quality is reduced to about 70%.

Similarly, Fig. 4(d,e,f) shows the effects of pairs of tactics combined. Infiltration is dominant, but more harm can be done in combination with flooding or deception: the average quality is reduced to 40% when $\gamma=0.1$ and $\theta=32$ (Fig. 4(d)) or $\phi=1$ (Fig. 4(e)). Combining flooding and deception ($\theta=32,\phi=1$, Fig. 4(f)) only results in marginal loss of quality (below 70%). With all three tactics combined ($\gamma=0.1,\theta=32,\phi=1$, not shown), bad actors can further reduce the quality to 30%.

Reshare and Exposure Cascades

Empirical evidence has shown that among fact-checked claims, low-quality content (debunked claims) tends to have larger retweet cascades than high-quality content (confirmed claims) (9, 41). SimSoM allows us to examine how different factors may contribute to such a virality pattern. Fig. 5 illustrates the effects of bad actor tactics on the size of reshare cascades for content generated by both authentic agents (“high-quality”) and inauthentic agents (“low-quality”). We observe that low-quality content can be boosted most effectively through high bad actor infiltration. This appears to be at odds with the hypothesis that attributes the empirical difference in virality to factors like novelty, which make false news more appealing, rather than to inauthentic accounts such as social bots (9). In our model, even the highest boost in appeal ($\phi=1$) does not grant low-quality cascades a virality comparable to those under higher bad actor amplification. A couple of observations reconcile the apparent contradiction. First, multiple factors may contribute to the virality of low-quality information, including inauthentic accounts and deceptively appealing content. Second, data from Twitter does not allow for the reconstruction of actual cascades. Therefore, empirical analyses may underestimate intermediary amplification by inauthentic accounts even when those accounts are removed. On the other hand, our model lets us reconstruct the likely reshare cascades that include intermediary amplification by inauthentic accounts.

To date, empirical data from social media platforms has allowed researchers to measure reshare cascades, but not exposure. Using the SimSoM model, it is possible to reconstruct not only likely reshare networks but also exposure networks, thus estimating how many accounts are exposed to (i.e., view) a message even if they do not reshare it (see Methods). Fig. 6 compares the sizes of reshare and exposure cascades. In general, reshares underestimate exposures by roughly one order of magnitude. Excluding the smallest and largest cascades, we observe that exposure networks grow sub-linearly with reshare networks: $s_{v}\sim s_{r}^{\nu}$ where $s_{v}$ and $s_{r}$ are the exposure and reshare cascade sizes, respectively, and $\nu<1$ is the scaling exponent. This means that as messages go viral, exposures grow more slowly than reshares. The exponent is higher for low-quality ($\nu\approx 0.8$) than for high-quality content ($\nu\approx 0.6$), suggesting that for each extra reshare, messages posted by inauthentic agents gain more views.

Targeting Tactics

The above results show that bad actors can use inauthentic accounts to infiltrate and disrupt an online public square. We have thus far assumed that all authentic agents have the same probability of following inauthentic ones, reflecting a scenario in which bad actors do not focus their efforts on specific potential followers. However, those interested in manipulating the network may want to maximize the spread of low-quality content through the community by targeting certain groups of accounts.

As an example, an adversary might target influentials based on the assumption that having such followers can multiply their impact — a message reshared by an influential account has a higher chance of going viral. Targeting influentials is well within the capability of bad actors and even automated accounts; the number of followers, often used as a proxy for influence (42), is public information on all social media platforms. A bad actor can easily interact with accounts having many followers by mentioning and/or following them (43, 44); other ploys include retweeting, quoting, and/or liking their tweets. There is empirical evidence of preferential targeting by bad actors that spread misinformation (1, 8).

Targeting politically active accounts or habitual misinformation spreaders are also conceivable tactics. An important question, then, is whether tactics targeting specific authentic accounts do in fact increase the manipulative power of bad actors. To explore this question, we introduce a preferential targeting tactic for adversarial actors in the model. Authentic agents have different probabilities of following inauthentic ones, proportional to one of five features that are available in the empirical data: number of followers (hubs tactic), propensity to share misinformation (misinformation tactic), political partisanship (partisanship tactic), or specific political leaning (liberal and conservative tactic). See Methods for details.

Fig. 7 shows the impact of these targeting tactics on information quality. Counterintuitively, preferential targeting is less harmful than random targeting: the distribution of quality is uneven so that the targeted population is worse off, but other parts of the community are spared. Targeting tactics therefore tend to backfire if we assume that bad actors intend to maximize the spread of their content across the full community.

Targeting hubs, for example, results in a network with significantly higher average quality ($p<10^{-4}$). The amplification power of an influential is counterbalanced by the concentration of low-quality content, which has fewer chances to be reshared; the majority of other agents are left relatively free from manipulation. Note that this is analog to the reason why the presence of hubs leads to high-quality content being forgotten more quickly when hubs are not targeted by bad actors, as seen earlier.

The harm of manipulation through the network also diminishes when bad actors target accounts sharing a lot of misinformation ($p<10^{-3}$) or with specific political leaning ($p<10^{-4}$). The echo-chamber structure of the empirical network (Fig. 7(b)) helps interpret the latter finding: low-quality messages get shared and become obsolete rapidly within one densely connected partisan sub-community, sparing the rest of the network.

Discussion

Social media platforms have enhanced the so-called attention economy, in which abundant content must compete for our scarce attention (45). But how to ensure that accurate, relevant, timely information wins this competition? To date, the policies that govern social media have been mainly guided by the concept of a free marketplace of ideas, rooted in John Milton’s centuries-old reasoning that truth prevails in a free and open encounter of opinions (46). In an ideal world, the wisdom of the crowd (47) would realize this vision by combining the opinions of many users (48). Unfortunately, several aspects of modern social media challenge the illusion of a public square or marketplace in which wise crowds access and select quality information (49, 50). First, information production is affected by information consumption (51), creating incentives to produce appealing but not necessarily high-quality content. Second, social influence undermines the wisdom of crowds because the information and opinions to which we are exposed online may be inauthentic, correlated through coordination (52, 53), or dominated by few influential individuals (54). Third, confirmation bias (55) can increase vulnerability to misinformation in social media (56, 11). Finally, the structure of information flow networks can distort perceptions and increase vulnerability to malicious actors (57, 58).

Exploration using SimSoM quantifies how manipulation by bad actors can prevent social media from functioning as a public town square. Simulations of the model reveal that making low-quality content highly appealing plays a lesser role compared to other harmful tactics by inauthentic accounts. This suggests that novelty, for example, may not provide the primary explanation for the virality of fake news, as previously hypothesized (9). Tactics that inflate engagement indicators, such as flooding, can erode the system’s quality just as well. More importantly, we find that infiltrating the network is a dominant harmful tactic available to bad actors — these accounts can drastically suppress quality by inducing only a small fraction of the community into following them.

A wealth of previous models and experiments have focused on information diffusion and popularity. Several studies have investigated the role played by network mechanisms affecting the popularity of individual posts, including exogenous and endogenous bursts of attention (59, 60), memory (61), novelty (62, 63), and position bias (64, 65). This literature considers the popularity of pieces of information in isolation. Market-like environments in which many messages compete for limited attention have received less consideration. Exceptions have considered the cost of learning about quality (66), distortions of quality assessments that result from aggregate knowledge of peer choices (52), and confirmation bias in the spread of misinformation (67). The SimSoM model proposed here extends the model of Weng et al. (68), who demonstrated that some posts inevitably achieve viral popularity irrespective of quality in the presence of competition among networked agents with limited attention. The model was formalized as a critical branching process and studied analytically, predicting that the popularity of posts follows a power-law distribution with heavy tails (69, 70, 71).

The current SimSoM model has several limitations. First, it assumes that all authentic agents are similar, when in reality users have diverse behaviors. For example, in the U.S., conservative social media users are more likely to share low-credibility content (7, 72, 44). Large language models have recently been proposed as a way to model agents with more realistic and heterogeneous behaviors (73). Second, SimSoM models information diffusion through simple contagion. While this approach is supported by empirical evidence (41, 74), complex contagion (25) can also play a role in the spread of certain types of harmful information (26). Finally, SimSoM neglects many mechanisms of actual online social media that may contribute to message exposure, such as search and the secret algorithmic details of each platform. Yet the model captures several universal ingredients of socio-technical networks, including selection criteria (appeal, social engagement, and recency) and limited individual attention. The empirical follower network also accounts for structural features (hubs, clustering, polarized communities) that play a key role in information diffusion (75, 76, 72, 77, 78). The model’s predictions are consistent with empirical findings about the difference in virality or lack thereof between low- and high-quality information on Twitter (9, 8). This suggests a reasonable balance between model realism and generality.

We can think of inauthentic accounts as zealots, a minority of agents committed to a particular view. Opinion dynamics models have shown that a critical minority of active zealots can quickly drive a system to consensus toward their opinion (79, 80, 81, 82). In our social media model, we only explore the capacity to suppress information quality rather than to drive consensus to a particular opinion.

Our results suggest that, surprisingly, inauthentic accounts do not need to target hubs (influentials); they can do more damage by connecting to random accounts. Future research should focus on whether the strategic placement of bad actors within/across polarized online communities can sway users toward a particular outcome, for example by distorting popularity perceptions (57, 58). Further work is also needed to characterize the effects of strategic targeting when the attacker has limited resources.

The insights gained from the present findings suggest several countermeasures to increase the resilience of social media to manipulation. The first lesson is that we must make it more difficult for bad actors to infiltrate the network. Platform efforts to detect and take down deceptive accounts must be strengthened, especially as tactics to hack follower networks get more sophisticated (43).

Recent models show that caps on the depth and/or breadth of diffusion networks can decrease the ratio of distorted messages received by social media users (83). While these models assume that distortions occur randomly, SimSoM demonstrates how adversarial actors can exploit vulnerabilities by flooding our news feeds, thereby crowding out quality information. A countermeasure would be to challenge accounts that post at very high rates to prove that they are human. Users could also be warned when they follow accounts that post low-quality content and/or when friend accounts are suspended or take suspicious actions, such as changing names/handles (14) or posting and deleting large volumes of content (40).

Our findings suggest that a winning tactic for malicious accounts is to target all social media users, and not only the most influential, if their goal is to spread harm widely across the network. Literacy programs may provide some protection against disinformation. This is also suggested by a binary agreement model that assumes each agent may be committed to truth ordisinformation (84). Social media platforms could lead by educating users about their vulnerability to manipulation and deception. The simplest version of this is through accuracy reminders, which can improve content quality during both posting (85) and resharing (86). Flooding reduces the effectiveness of such accuracy interventions (87). Our model could be extended to explore ways to combine friction and accuracy nudges by, e.g., lowering the probability that low-quality messages are reshared by agents after they are exposed to warnings.

Given that inauthentic accounts can be used to suppress human speech, granting them (or any entity that controls them) unlimited free-speech rights would seem to lead to a logical contradiction (88). Yet, ironically, efforts by social media platforms to moderate abusive accounts and even research on social bot detection have been assailed by some with charges of censorship (89). These questions may have significant repercussions on regulations designed to protect the online public square (49).

Methods

Social Media Diffusion Model

SimSoM111Code and data to implement the model and reproduce results are available at github.com/osome-iu/SimSoM is a parsimonious agent-based model inspired by the long tradition of representing the spread of ideas as an epidemic process where messages are passed along the edges of a network (90). The model simulates a directed follower network, as in Twitter/X, Mastodon, or Threads. Nodes represent agents (users) and links represent follower relations, which may or may not be reciprocal. The direction of a link goes from the follower to the followed (friend) account, capturing the flow of attention; when a friend’s post is reshared by a follower, information spreads in the opposite direction. In line with previous work (74, 41, 91), the diffusion process is modeled as simple contagion, where each exposure to a message results in the same resharing probability. We assume that the structure of the network is static (no unfollowing/blocking of accounts) during the information-spreading process. In contrast to classical epidemiological models, new messages are continuously introduced into the system in an exogenous fashion.

At each time step, an agent $i$ produces a new message with probability $\mu$ or chooses one of the messages in their news feed to be reshared with probability $1-\mu$ (Fig. 1). The new or reshared message is then added to the news feeds of $i$’s followers. We set $\mu=0.5$, reflecting the empirical average ratio for English-language tweets (92). Based on empirical data, agents are assumed to have limited-size inventories: only the most recent $\sigma=15$ messages are retained in each news feed (see Supplementary Material). The results are robust for different values of $\mu$ and $\sigma$ (see Fig. S1 in Supplementary Material).

We assume authentic agents prefer to reshare messages posted by their friends that are appealing, recent, and popular. This is based on empirical evidence that users are more likely to share popular content according to engagement signals (93). The probability that an agent shares a message from their news feed, allowing it to spread, is proportional to the message’s appeal, social engagement, and recency. More explicitly, let $M_{i}$ be the feed of $i$ ($|M_{i}|=\sigma$). The probability of message $m\in M_{i}$ being selected is $P(m)=a_{m}e_{m}r_{m}/\sum_{j\in M_{i}}a_{j}e_{j}r_{j}$ where $a_{m}$ is the appeal of message $m$, $e_{m}$ is the social engagement, i.e., the number of times it was (re)shared by $i$’s friends, and $r_{m}$ is the recency. Message recency decays with time as a stretched exponential function $r_{m}(t)=e^{-0.4t^{0.4}}$, where $t$ is the “age,” or the time passed since $m$ was first introduced to the agent’s news feed. This decay function is based on empirical online news engagement data (63). To model flooding, content from bad actors has exposure that is $\theta$ times higher than authentic content.

Quality and Appeal

Both the quality $q$ and the appeal $a$ of new messages are defined in the unit interval. Informed by empirical data, $q$ and $a$ for authentic accounts are assumed to be independent (see Fig. S2 in Supplementary Material). For authentic accounts, quality $q$ is drawn from an exponential distribution $P(q)=Ce^{-\tau q}$ where high-quality information is more common than low-quality information. The term $C=\frac{\tau}{1-e^{-\tau}}$ is a normalizing constant such that $\int_{0}^{1}P(q)dq=1$; the exponent $\tau=-10$ is estimated empirically (see Fig. S3 in Supplementary Material). We independently draw appeal from the distribution $P(a)=(1+\alpha)(1-a)^{\alpha}$, where $\alpha>1$ captures the rarity of appealing messages (Fig. S4(a) in Supplementary Material). We set $\alpha=4$. This choice for the appeal probability density function reproduces a broad distribution of reshares comparable to that observed in real-world information diffusion networks: few messages go viral while the majority do not (Fig. S4(b) in Supplementary Material).

On the other hand, we assume that bad actors strictly generate low-quality messages ($q=0$). The potentially deceptive nature of this content is modeled by the deception parameter $\phi$ ($0\leq\phi\leq 1$), the probability that a bad actor message is irresistibly appealing. With probability $\phi$, we set $a=1$, and with probability $1-\phi$ we draw appeal from the same distribution as for authentic accounts, $P(a)=(1+\alpha)(1-a)^{\alpha}$. If $\phi=0$, bad actors and authentic accounts generate messages with appeal drawn from the same distribution. If $\phi>0$, bad actor messages are more likely to have high appeal; the larger $\phi$, the greater the potential virality of low-quality content by bad actors.

Empirical Follow Network

We run simulations on a follower network derived from empirical Twitter data. This network was constructed from a 10% random sample of public tweets between June 1–30, 2017 (94). The data includes users (excluding likely automated accounts) that shared at least ten links to news sources, at least one of which was to a source labeled as low-quality. Only news sources with known political valence were considered. This sampling procedure captures a community of accounts that are both active and vulnerable to misinformation. Based on the shared links, most accounts in the dataset have an associated partisanship score (from $-1$ for left-leaning to $+1$ for right-leaning) defined as the average political bias of the news sources they share; and a misinformation score defined as the fraction of posts linking to low-credibility sources. Both partisanship and misinformation scores, included in the original network dataset, were based on news source labels from third-party fact-checkers (72).

From the original dataset, we select nodes with both partisanship and misinformation attributes. We further reduce the size of the network to speed up our simulations. We apply $k$-core decomposition to select $N=10{,}006$ nodes forming the $k=94$ core. Finally, we remove a random sample of edges to decrease the density of this core while preserving the average in/out-degree (number of friends/followers) of the original network ($k=0$ core). This results in $E=1{,}809{,}798$ edges; each node has on average approximately 180 friends/followers (Fig. 3, “hubs+communities” network).

Bad Actor Subnetwork

Since the empirical network described above does not include likely inauthentic accounts, we use it to model the subnetwork of authentic accounts. We then add a subnetwork representing inauthentic accounts that infiltrate the system. The ratio between the sizes of the two subnetworks is described by $\beta$, i.e., for an authentic agent subnetwork of $N$ nodes, the inauthentic subnetwork is composed of $\beta N$ nodes (Fig. 2). Since there are many types of inauthentic accounts (trolls, social bots, cyborgs), estimating the percentage of these on social media is a very difficult task. We thus set $\beta=0.05$ following a rough estimation of the prevalence of bots on Twitter.222theconversation.com/how-many-bots-are-on-twitter-the-question-is-difficult-to-answer-and-misses-the-point-183425 Note that from the perspective of information quality in the model, increasing the prevalence of bad actors is equivalent to increasing their infiltration. Therefore we focus on the effects of varying $\gamma$ rather than $\beta$.

The model assumes that inauthentic accounts follow each other to amplify low-quality content. To capture the characteristic presence of hubs and clustering (directed triads), the inauthentic subnetwork is created using a directed variant of the random-walk growth model (95). Specifically, the network is initialized with four fully connected nodes. We then add one new node at a time, assuming that each has fixed out-degree $k_{out}=3$. Once a new node $i$ comes into the network, it links to (follows) a randomly selected target node (friend) $j$. Each of the remaining $k_{out}-1=2$ friends are selected as follows: with probability expressed by a parameter $p$, $i$ follows a random friend of $j$’s; with probability $1-p$, $i$ follows another randomly selected node. Following friends of a friend has the effect of generating closed, directed triads and approximates a preferential attachment process, giving rise to hub nodes with high in-degree. The parameter $p$ thus models both hubs and clustering. We use $p=0.5$.

In addition, the bad actor subnetwork is designed to manipulate information flow and collective attention by spreading certain messages. Therefore we assume that bad actors get random authentic accounts to follow them: we add a directed link from each authentic node to each bad actor node with probability $\gamma$. (In the next subsection we present alternatives to this random targeting tactic.) The parameter $\gamma$ models the degree of infiltration of the network by bad actors (Fig. 2). When $\gamma=0$, there is no infiltration and bad actors are isolated, therefore harmless; the opposite extreme $\gamma=1$ indicates complete infiltration such that bad actors dominate the network. Because we are not concerned with the quality of messages on bad actor news feeds, they do not follow or reshare content from authentic agents for the analyses in this paper.

Bad Actor Targeting Tactics

In the scheme described above, the authentic agents that follow bad actors do so randomly. We also study scenarios in which bad actors target certain accounts as potential followers (Fig. 7). In all of these scenarios, each bad actor is still followed by $\gamma N$ authentic accounts on average, but these targets are selected according to some criterion, such as whether they are politically active or have many followers.

Each targeting tactic is modeled by making the probability that an authentic agent $i$ follows bad actors proportional to some feature $f(i)$. In the hubs tactic, we set $f(i)=k_{in}(i)$, the number of followers of $i$. Misinformation and partisanship attributes of authentic accounts in the empirical network allow us to model other targeting scenarios. In the misinformation tactic, $f(i)$ is set to $i$’s misinformation score, i.e., their propensity to share misinformation. The partisanship score is used in the liberal and conservative tactics, while its absolute value is used in the partisanship tactic.

Overall Quality

The effects of manipulation on authentic agents are quantified by the quality of all content in circulation. The overall quality of the information system at time $t$ is measured by the average quality across all the messages visible through the feeds of the authentic agents:

[TABLE]

where $q_{i,m,t}$ is the quality of the message in the $m$-th position in authentic user $i$’s feed at time $t$.

The system’s quality at each time step $t$ is calculated with an exponential moving average $\bar{Q}_{t}=\rho\bar{Q}_{t-1}+(1-\rho)Q_{t}$. As the simulation takes place, some of the messages become obsolete quickly, while others live longer and infect a larger fraction of the network. The simulation ends once the system reaches a steady state in which the difference between the average quality in two consecutive time steps is smaller than a threshold, i.e., $|\bar{Q}_{t}-\bar{Q}_{t-1}|/\bar{Q}_{t-1}<\epsilon$. The average quality reported for all analyses is calculated at the end of the simulation.

A reshare cascade is a tree that begins when an agent $i$ (root) posts a new message $m$. After that, a new link is created when a follower $j$ of $i$ reshares $m$. We say that $i$ is the parent of $j$ in the tree. Similarly, an agent’s exposure to a message is defined as having that message on their feed while being activated — we assume that an agent who is about to share or reshare a post has seen the content on their feed. A link in the exposure cascade for message $m$ is created between an activated agent with $m$ in their feed and their friend who had shared $m$. When constructing both reshare and exposure cascades, $m$ may have been reshared by more than one of an agent’s friends. When this occurs, one of them is selected at random to be the parent in the cascade tree. This choice does not affect a cascade tree’s size, but might affect its structure. Other alternatives, such as selecting as parent the friend who most recently shared $m$, can be explored if one desires to analyze the structure of the reshare or exposure cascades. By definition, reshare cascades can be as small as one and exposure cascades can be as small as zero.

To capture the complete diffusion cascades, the size distributions plotted in Fig. 5 include only cascades of “extinct” messages, those that have become obsolete and are no longer present in any news feed by the end of simulations.

Simulation Framework and Parameters

For each set of parameters, we run 10 simulations starting from random conditions; the reported average quality is the average across many runs.

In scenarios where there are bad actors in the system, the default bad actor parameters are $\gamma=0.01$, $\phi=0$, and $\theta=1$. Except for the simulations exploring inauthentic targeting tactics, authentic followers of the bad actors are selected at random.

The parameters $\rho=0.8$ and $\epsilon=0.0001$ were tested to ensure that the system’s quality stabilizes at the steady state.

We shuffle the empirical follower network in various ways to derive the scenarios reported in Fig. 3. The “hubs+communities” network is the original one. In the “hubs” network, we shuffle the original network while preserving the degree distribution and destroying any community clustering. In the “communities” network, the outgoing endpoint of each edge in the original network is rewired to another node within the same community; this preserves the community structure and destroys the hub structure. In the “random” shuffle, all edges are rewired at random with uniform probability, destroying all hub and clustering structure.

Acknowledgments

We are grateful to Marshall Van Alstyne, David Axelrod, Rachith Aiyappa and Erfan Samieyan for useful discussion and suggestions; to Dimitar Nikolov, Chengcheng Shao, Giovanni Ciampaglia, and Pik-Mai Hui for the data collection used to construct the empirical network; to Kai-Cheng Yang and Christopher Torres-Lugo for the COVID-19 data collection; and to Alireza Sahami Shirazi for data about scrolling session depth on a social media mobile app. This work was carried out in part while XL visited the Observatory on Social Media at Indiana University, with support by the China Scholarship Council. BT, AF and FM were supported in part by The Knight Foundation. BT was funded in part by the Ostrom Workshop. AF and FM were funded in part by DARPA (grants W911NF-17-C-0094 and HR001121C0169). FM and BT were supported in part by the Swiss National Science Foundation (Sinergia grant CRSII5_209250). FM was supported in part by Democracy Fund and Craig Newmark Philanthropies. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.

Supplementary Material

S1 News feed size and information load

We use a news feed size $\sigma=15$. This value is approximated from the average depth of approximately 107 mobile scrolling sessions measured on Tumblr during two weeks in 2016. The feed interface of this app was similar to those of other social media platforms. We considered a session to have ended when there was no interaction for 30 minutes or longer. The depth of a session was recorded as the number of times that a user scrolled at least 500 pixels through the feed and then stopped for at least one second.

Results are robust for different values of news feed size $\sigma$ as well as posting activity $\mu$ (Fig. S1).

S2 Relationship between quality and engagement

To estimate the relationship between quality and engagement, we start from a dataset of tweets about COVID-19 (96). The dataset consists of tweets containing the hashtags #coronavirus or #covid19 collected in 2020, selected from a 10% random sample of public tweets (97). From this set of tweets, we analyze only posts that shared links to news sources from March 9–29, 2020. Linked sources were extracted from URLs in the tweet metadata. URLs shortened with 70 popular URL shortening services were also expanded to reveal the actual sources. Posts with links to Twitter and other social media platforms were excluded. The remaining links were matched against sources with ratings obtained from NewsGuard333newsguardtech.com in April 2021. The final dataset contains 110,224 original Twitter posts that share at least one link with a NewsGuard rating. For each post we have both the rating of the shared source, used as a proxy for post quality, and the number of retweets. The latter is extracted from the metadata of the latest retweet of each post.

Fig. S2 shows that the engagement of a post, as measured by its retweet count, is weakly correlated with its quality (Spearman coefficient $-0.13$). Based on this observation, we assume that appeal is independent of quality in the model.

S3 Empirical estimation of quality distribution

The same empirical data described above is used to estimate the distribution of authentic message quality. The quality of a post is the average Newsguard score of the news sources shared in the post, normalized to be in the unit interval. As shown in Fig. S3, we fit the data to the exponential probability density function $P(q)=\frac{\tau}{1-e^{-\tau}}e^{-\tau q}$, with $\tau=-10$. The normalization constant is obtained by setting $\int_{0}^{1}P(q)dq=1$.

S4 Empirical estimation of appeal distribution

The appeal distribution of authentic content is modeled by the probability density function $P(a)=(1+\alpha)(1-a)^{\alpha}$. The normalization constant is obtained by setting $\int_{0}^{1}P(a)da=1$. This captures the intuition that most messages have low appeal. As illustrated in Fig. S4(a), high appeal values are much more rare for $\alpha=10$ than $\alpha=1$. While there is no empirical proxy data for appeal, a reasonable choice should give rise to a distribution of engagement consistent with empirical reshare data. Fig. S4(b) shows that values $\alpha\geq 1$ result in distributions of engagement in the model that roughly capture the broad distribution of reshares in the same empirical data described above.