Applied Theory of Mind and Large Language Models -- how good is ChatGPT at solving social vignettes?

TL;DR

This study evaluates GPT-3.5 Turbo and GPT-4's ability to perform social reasoning tasks, finding GPT-4 performs near human levels on complex Theory of Mind assessments, indicating potential for assistive social communication tools.

Contribution

It provides a comprehensive assessment of GPT-4's capacity for applied Theory of Mind in social vignettes, comparing it to human performance and previous AI models.

Findings

GPT-4 achieves near human accuracy on the Faux Pas Test.

GPT-4 scores comparable to neurotypical adults on Social Stories Questionnaire.

GPT-4 exceeds neurotypical benchmarks in Story Comprehension Test.

Abstract

The rapid development of language-based artificial intelligence (AI) offers new possibilities for psychotherapy and assistive systems, particularly benefitting autistic individuals who often respond well to technology. Parents of autistic persons emphasize the importance of appropriate and context-specific communication behavior. This study investigated whether GPT-3.5 Turbo and GPT-4, as language-based AI applications, are fundamentally capable of replicating this type of adequate communication behavior in the form of applied Theory of Mind (ToM). GPT-3.5 Turbo and GPT-4 were evaluated on three established higher-order ToM tasks: the Faux Pas Test, the Social Stories Questionnaire, and the Story Comprehension Test in English and German. Two independent raters scored response accuracy based on standardized manuals. In addition, responses were rated for epistemic markers as indicators of uncertainty. GPT's results were compared to human neurotypical and neurodivergent samples from previous own and others' research. GPT-4 achieved near human accuracy on the Faux Pas Test and outperformed GPT-3.5 Turbo and individuals with autistic traits. On the Social Stories Questionnaire, GPT-4 scored comparable to neurotypical adults, while GPT-3.5 Turbo remained well below. In the Story Comprehension Test, GPT-4 reached scores that exceeded neurotypical adult and adolescent benchmarks. However, GPT-4 used epistemic markers in up to 42% of responses. GPT-4 shows encouraging performance in complex higher-order ToM tasks and may offer future potential as an assistive tool for individuals with (and without) social communication difficulties. Its ability to interpret complex social situations is promising; however, the frequent use of uncertainty markers highlights the need for further study for assistive use and possibly further refinement to ensure consistent and reliable support in real-world use.