LLMs Struggle with Abstract Meaning Comprehension More Than Expected
Hamoud Alhazmi, Jiachen Jiang

TL;DR
This paper evaluates large language models' ability to understand abstract meanings, revealing significant challenges and proposing a bidirectional attention classifier that improves performance.
Contribution
It highlights the difficulty LLMs face with abstract concepts and introduces a novel attention-based method that enhances model accuracy in this task.
Findings
Most LLMs struggle with abstract meaning comprehension in zero-, one-, and few-shot settings.
Fine-tuned models outperform LLMs in understanding abstract concepts.
The proposed bidirectional attention classifier improves accuracy by over 3%.
Abstract
Understanding abstract meanings is crucial for advanced language comprehension. Despite extensive research, abstract words remain challenging due to their non-concrete, high-level semantics. SemEval-2021 Task 4 (ReCAM) evaluates models' ability to interpret abstract concepts by presenting passages with questions and five abstract options in a cloze-style format. Key findings include: (1) Most large language models (LLMs), including GPT-4o, struggle with abstract meaning comprehension under zero-shot, one-shot, and few-shot settings, while fine-tuned models like BERT and RoBERTa perform better. (2) A proposed bidirectional attention classifier, inspired by human cognitive strategies, enhances fine-tuned models by dynamically attending to passages and options. This approach improves accuracy by 4.06 percent on Task 1 and 3.41 percent on Task 2, demonstrating its potential for abstract…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
