Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

TL;DR

This paper evaluates large language models' ability to understand phrase semantics using human-annotated datasets, comparing prompting techniques and analyzing limitations, revealing they outperform traditional methods but have notable challenges.

Contribution

It provides a comprehensive assessment of LLMs' phrase semantic reasoning capabilities and analyzes the effects of different prompting strategies and their limitations.

Findings

LLMs outperform traditional embedding methods in phrase semantics tasks.

Advanced prompting strategies show variable effectiveness.

Error analysis highlights key limitations in LLM understanding.

Abstract

Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics.