Structured Event Reasoning with Large Language Models

TL;DR

This paper introduces three structured approaches combining large language models with symbolic representations to improve complex event reasoning, interpretability, and performance across diverse NLP tasks.

Contribution

It proposes three novel methods integrating LLMs with structured event representations, enhancing reasoning accuracy and interpretability over traditional end-to-end LLM approaches.

Findings

All three approaches outperform end-to-end LLMs on event reasoning tasks.

Structured representations improve interpretability of reasoning processes.

Semi-symbolic and symbolic methods show significant gains in complex reasoning scenarios.

Abstract

Reasoning about real-life events is a unifying challenge in AI and NLP that has profound utility in a variety of domains, while fallacy in high-stake applications could be catastrophic. Able to work with diverse text in these domains, large language models (LLMs) have proven capable of answering questions and solving problems. However, I show that end-to-end LLMs still systematically fail to reason about complex events, and they lack interpretability due to their black-box nature. To address these issues, I propose three general approaches to use LLMs in conjunction with a structured representation of events. The first is a language-based representation involving relations of sub-events that can be learned by LLMs via fine-tuning. The second is a semi-symbolic representation involving states of entities that can be predicted and leveraged by LLMs via few-shot prompting. The third is a fully symbolic representation that can be predicted by LLMs trained with structured data and be executed by symbolic solvers. On a suite of event reasoning tasks spanning common-sense inference and planning, I show that each approach greatly outperforms end-to-end LLMs with more interpretability. These results suggest manners of synergy between LLMs and structured representations for event reasoning and beyond.