Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation

TL;DR

This paper explores whether large language models understand the thematic fit of event arguments, demonstrating state-of-the-art results and analyzing differences between closed and open models in reasoning and filtering capabilities.

Contribution

It introduces novel prompting strategies to evaluate LLMs' knowledge of thematic fit and compares their performance, revealing distinct behaviors between closed and open models.

Findings

Closed models outperform open models in overall thematic fit scoring.

Multi-step reasoning improves closed models' performance.

Closed models are less effective at filtering incompatible sentences.

Abstract

The thematic fit estimation task measures semantic arguments' compatibility with a specific semantic role for a specific predicate. We investigate if LLMs have consistent, expressible knowledge of event arguments' thematic fit by experimenting with various prompt designs, manipulating input context, reasoning, and output forms. We set a new state-of-the-art on thematic fit benchmarks, but show that closed and open weight LLMs respond differently to our prompting strategies: Closed models achieve better scores overall and benefit from multi-step reasoning, but they perform worse at filtering out generated sentences incompatible with the specified predicate, role, and argument.