The Voice: Lessons on Trustworthy Conversational Agents from "Dune"

TL;DR

This paper discusses the risks of generative AI models being used for manipulation and influence, highlighting the need for strategies to prevent and detect such sociotechnical threats inspired by the 'Voice' concept from 'Dune'.

Contribution

It draws an analogy from 'Dune' to explore how generative AI can be weaponized for influence, emphasizing the importance of developing prevention and detection strategies.

Findings

Generative AI models can manipulate text, images, speech, and video.

These models are becoming affordable and widely accessible.

Potential for malicious use in shaping public opinion and undermining organizations.

Abstract

The potential for untrustworthy conversational agents presents a significant threat for covert social manipulation. Taking inspiration from Frank Herbert's "Dune", where the Bene Gesserit Sisterhood uses the Voice for influence, manipulation, and control of people, we explore how generative AI provides a way to implement individualized influence at industrial scales. Already, these models can manipulate communication across text, image, speech, and most recently video. They are rapidly becoming affordable enough for any organization of even moderate means to train and deploy. If employed by malicious actors, they risk becoming powerful tools for shaping public opinion, sowing discord, and undermining organizations from companies to governments. As researchers and developers, it is crucial to recognize the potential for such weaponization and to explore strategies for prevention, detection, and defense against these emerging forms of sociotechnical manipulation.