EvEntS ReaLM: Event Reasoning of Entity States via Language Models

TL;DR

This paper evaluates how well large language models understand entity state changes and physical attributes, revealing their limitations and demonstrating that proper prompting significantly enhances their reasoning abilities, especially in out-of-domain scenarios.

Contribution

The paper introduces a novel prompting technique that improves LLMs' reasoning about entity states and physical attributes, highlighting the importance of task encoding.

Findings

Proper prompting dramatically improves LLM performance.

LLMs struggle with reasoning about unseen attributes.

Prompting is especially effective with limited data.

Abstract

This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.