When Machines Get It Wrong: Large Language Models Perpetuate Autism Myths More Than Humans Do

TL;DR

This study reveals that current large language models more frequently perpetuate myths about Autism Spectrum Disorder than humans, highlighting a critical need for improved AI accuracy and responsible deployment in health information.

Contribution

It provides empirical evidence that leading AI systems currently propagate more misconceptions about autism than humans, challenging assumptions about AI's superior knowledge.

Findings

Humans made fewer autism myth errors than AI models.

AI models failed on 18 of 30 autism-related items compared to humans.

Results highlight a significant blind spot in AI systems' understanding of autism.

Abstract

As Large Language Models become ubiquitous sources of health information, understanding their capacity to accurately represent stigmatized conditions is crucial for responsible deployment. This study examines whether leading AI systems perpetuate or challenge misconceptions about Autism Spectrum Disorder, a condition particularly vulnerable to harmful myths. We administered a 30-item instrument measuring autism knowledge to 178 participants and three state-of-the-art LLMs including GPT-4, Claude, and Gemini. Contrary to expectations that AI systems would leverage their vast training data to outperform humans, we found the opposite pattern: human participants endorsed significantly fewer myths than LLMs (36.2% vs. 44.8% error rate; z = -2.59, p = .0048). In 18 of the 30 evaluated items, humans significantly outperformed AI systems. These findings reveal a critical blind spot in current AI systems and have important implications for human-AI interaction design, the epistemology of machine knowledge, and the need to center neurodivergent perspectives in AI development.