LLMs Struggle with Abstract Meaning Comprehension More Than Expected

TL;DR

This paper evaluates large language models' ability to understand abstract meanings, revealing significant challenges and proposing a bidirectional attention classifier that improves performance.

Contribution

It highlights the difficulty LLMs face with abstract concepts and introduces a novel attention-based method that enhances model accuracy in this task.

Findings

Most LLMs struggle with abstract meaning comprehension in zero-, one-, and few-shot settings.

Fine-tuned models outperform LLMs in understanding abstract concepts.

The proposed bidirectional attention classifier improves accuracy by over 3%.

Abstract

Understanding abstract meanings is crucial for advanced language comprehension. Despite extensive research, abstract words remain challenging due to their non-concrete, high-level semantics. SemEval-2021 Task 4 (ReCAM) evaluates models' ability to interpret abstract concepts by presenting passages with questions and five abstract options in a cloze-style format. Key findings include: (1) Most large language models (LLMs), including GPT-4o, struggle with abstract meaning comprehension under zero-shot, one-shot, and few-shot settings, while fine-tuned models like BERT and RoBERTa perform better. (2) A proposed bidirectional attention classifier, inspired by human cognitive strategies, enhances fine-tuned models by dynamically attending to passages and options. This approach improves accuracy by 4.06 percent on Task 1 and 3.41 percent on Task 2, demonstrating its potential for abstract meaning comprehension.