Modes of Analyzing Disinformation Narratives With AI/ML/Text Mining to Assist in Mitigating the Weaponization of Social Media

TL;DR

This paper discusses the development of AI/ML-based methods for analyzing and monitoring malicious disinformation narratives on social media, emphasizing the need for automated tools to combat increasing mal-info and generative AI misuse.

Contribution

It introduces initial text mining approaches for characterizing malicious information and explores the potential risks of generative AI in spreading disinformation.

Findings

Preliminary text analysis of mal-info posts on unmoderated platforms

Identification of inflammatory terms and patterns in malicious narratives

Discussion of generative AI's potential to amplify disinformation risks

Abstract

This paper highlights the developing need for quantitative modes for capturing and monitoring malicious communication in social media. There has been a deliberate "weaponization" of messaging through the use of social networks including by politically oriented entities both state sponsored and privately run. The article identifies a use of AI/ML characterization of generalized "mal-info," a broad term which includes deliberate malicious narratives similar with hate speech, which adversely impact society. A key point of the discussion is that this mal-info will dramatically increase in volume, and it will become essential for sharable quantifying tools to provide support for human expert intervention. Despite attempts to introduce moderation on major platforms like Facebook and X/Twitter, there are now established alternative social networks that offer completely unmoderated spaces. The paper presents an introduction to these platforms and the initial results of a qualitative and semi-quantitative analysis of characteristic mal-info posts. The authors perform a rudimentary text mining function for a preliminary characterization in order to evaluate the modes for better-automated monitoring. The action examines several inflammatory terms using text analysis and, importantly, discusses the use of generative algorithms by one political agent in particular, providing some examples of the potential risks to society. This latter is of grave concern, and monitoring tools must be established. This paper presents a preliminary step to selecting relevant sources and to setting a foundation for characterizing the mal-info, which must be monitored. The AI/ML methods provide a means for semi-quantitative signature capture. The impending use of "mal-GenAI" is presented.