As Good As A Coin Toss: Human detection of AI-generated images, videos, audio, and audiovisual stimuli

TL;DR

This study reveals that people currently struggle to reliably distinguish AI-generated media from real content, especially with more convincing synthetic media, highlighting the need for alternative detection methods.

Contribution

The paper provides empirical evidence that human perceptual detection of AI-generated media is near chance level, emphasizing the urgency for automated detection solutions.

Findings

Participants' detection accuracy was close to 50%, near chance level.

Accuracy decreases with synthetic content, foreign languages, and single modality media.

Older individuals perform worse at detecting synthetic media.

Abstract

One of the current principal defenses against weaponized synthetic media continues to be the ability of the targeted individual to visually or auditorily recognize AI-generated content when they encounter it. However, as the realism of synthetic media continues to rapidly improve, it is vital to have an accurate understanding of just how susceptible people currently are to potentially being misled by convincing but false AI generated content. We conducted a perceptual study with 1276 participants to assess how capable people were at distinguishing between authentic and synthetic images, audio, video, and audiovisual media. We find that on average, people struggled to distinguish between synthetic and authentic media, with the mean detection performance close to a chance level performance of 50%. We also find that accuracy rates worsen when the stimuli contain any degree of synthetic content, features foreign languages, and the media type is a single modality. People are also less accurate at identifying synthetic images when they feature human faces, and when audiovisual stimuli have heterogeneous authenticity. Finally, we find that higher degrees of prior knowledgeability about synthetic media does not significantly impact detection accuracy rates, but age does, with older individuals performing worse than their younger counterparts. Collectively, these results highlight that it is no longer feasible to rely on the perceptual capabilities of people to protect themselves against the growing threat of weaponized synthetic media, and that the need for alternative countermeasures is more critical than ever before.