Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphology

TL;DR

This paper investigates whether large language models can understand implied discourse meanings, specifically exhaustivity implicatures of Korean morphology, and finds that encoding discourse meanings is more challenging than semantic features.

Contribution

It evaluates the capability of large language models to capture discourse-level meanings associated with Korean morphological markers, highlighting the difficulty in encoding such discourse implicatures.

Findings

Language models struggle more with discourse meanings than semantic features.

Discourse meanings of grammatical markers are harder to encode than discourse markers.

Large models show limited understanding of implied discourse meanings in Korean morphology.

Abstract

Markedness in natural language is often associated with non-literal meanings in discourse. Differential Object Marking (DOM) in Korean is one instance of this phenomenon, where post-positional markers are selected based on both the semantic features of the noun phrases and the discourse features that are orthogonal to the semantic features. Previous work has shown that distributional models of language recover certain semantic features of words -- do these models capture implied discourse-level meanings as well? We evaluate whether a set of large language models are capable of associating discourse meanings with different object markings in Korean. Results suggest that discourse meanings of a grammatical marker can be more challenging to encode than that of a discourse marker.