Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

TL;DR

This paper examines how well large language models understand the nuanced meanings of discourse particles like 'just', revealing their limited ability to grasp subtle semantic distinctions despite some recognition of broader categories.

Contribution

It provides a detailed analysis of LLMs' capacity to interpret the complex, polyfunctional nature of discourse particles, using expert-labeled data for evaluation.

Findings

LLMs can distinguish broad categories of 'just'

LLMs struggle with subtle semantic nuances

There is a gap in LLMs' understanding of discourse particles

Abstract

Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle "just" (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English "just", a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.