Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios

TL;DR

This paper assesses large language models' understanding of metaphor and sarcasm using a psychiatric screening tool, revealing improved metaphor comprehension with larger models but no similar gains for sarcasm, highlighting the need for specialized training.

Contribution

It introduces a novel evaluation of LLMs' nuanced social communication skills using a standardized psychiatric screening test.

Findings

Larger models show better metaphor understanding.

No significant improvement in sarcasm comprehension with increased model size.

Sarcasm comprehension may require emotionally grounded training strategies.

Abstract

Metaphors and sarcasm are precious fruits of our highly evolved social communication skills. However, children with the condition then known as Asperger syndrome are known to have difficulties in comprehending sarcasm, even if they possess adequate verbal IQs for understanding metaphors. Accordingly, researchers had employed a screening test that assesses metaphor and sarcasm comprehension to distinguish Asperger syndrome from other conditions with similar external behaviors (e.g., attention-deficit/hyperactivity disorder). This study employs a standardized test to evaluate recent large language models' (LLMs) understanding of nuanced human communication. The results indicate improved metaphor comprehension with increased model parameters; however, no similar improvement was observed for sarcasm comprehension. Considering that a human's ability to grasp sarcasm has been associated with the amygdala, a pivotal cerebral region for emotional learning, a distinctive strategy for training LLMs would be imperative to imbue them with the ability in a cognitively grounded manner.