ProRAC: A Neuro-symbolic Method for Reasoning about Actions with LLM-based Progression

TL;DR

ProRAC is a neuro-symbolic framework that uses large language models to reason about actions and change by extracting elements, executing actions progressively, and evaluating queries, demonstrating strong performance across various benchmarks.

Contribution

This work introduces ProRAC, a novel neuro-symbolic approach that leverages LLMs for reasoning about actions and change, integrating symbolic execution with language models.

Findings

Achieves strong performance on multiple RAC benchmarks

Effective across different domains and LLM backbones

Demonstrates versatility in various RAC task types

Abstract

In this paper, we propose ProRAC (Progression-based Reasoning about Actions and Change), a neuro-symbolic framework that leverages LLMs to tackle RAC problems. ProRAC extracts fundamental RAC elements including actions and questions from the problem, progressively executes each action to derive the final state, and then evaluates the query against the progressed state to arrive at an answer. We evaluate ProRAC on several RAC benchmarks, and the results demonstrate that our approach achieves strong performance across different benchmarks, domains, LLM backbones, and types of RAC tasks.