Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL

TL;DR

This paper evaluates the ability of Large Language Models to understand structured semantics through a Semantic Role Labeling task, revealing their potential and limitations, and comparing their errors to those of untrained humans.

Contribution

The study introduces PromptSRL, a few-shot prompting method for LLMs to perform SRL, providing insights into their semantic understanding capabilities and error patterns.

Findings

LLMs can capture semantic structures to some extent.

Scaling up LLMs does not always improve semantic understanding.

Both LLMs and untrained humans make similar errors in SRL.

Abstract

Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. Nevertheless, an ongoing controversy exists over the extent to which LLMs can grasp structured semantics. To assess this, we propose using Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to extract structured semantics. In our assessment, we employ the prompting approach, which leads to the creation of our few-shot SRL parser, called PromptSRL. PromptSRL enables LLMs to map natural languages to explicit semantic structures, which provides an interpretable window into the properties of LLMs. We find interesting potential: LLMs can indeed capture semantic structures, and scaling-up doesn't always mirror potential. Additionally, limitations of LLMs are observed in C-arguments, etc. Lastly, we are surprised to discover that significant overlap in the errors is made by both LLMs and untrained humans, accounting for almost 30% of all errors.