Full Automation of Goal-driven LLM Dialog Threads with And-Or Recursors and Refiner Oracles

TL;DR

This paper presents a method for fully automating goal-driven reasoning in LLM dialog threads by recursively exploring alternatives and details, guided by a logic-inspired algorithm and validation via semantic similarity and oracle advice.

Contribution

It introduces a recursive descent approach inspired by Horn Clause logic to automate deep reasoning in LLM dialogs, integrating natural language patterns and validation mechanisms.

Findings

Automated reasoning can be guided by semantic similarity to ground-truth facts.

The approach enables applications like causal explanations and scientific literature exploration.

The method achieves goal-focused dialog thread automation with minimal supervision.

Abstract

We automate deep step-by step reasoning in an LLM dialog thread by recursively exploring alternatives (OR-nodes) and expanding details (AND-nodes) up to a given depth. Starting from a single succinct task-specific initiator we steer the automated dialog thread to stay focussed on the task by synthesizing a prompt that summarizes the depth-first steps taken so far. Our algorithm is derived from a simple recursive descent implementation of a Horn Clause interpreter, except that we accommodate our logic engine to fit the natural language reasoning patterns LLMs have been trained on. Semantic similarity to ground-truth facts or oracle advice from another LLM instance is used to restrict the search space and validate the traces of justification steps returned as answers. At the end, the unique minimal model of a generated Horn Clause program collects the results of the reasoning process. As applications, we sketch implementations of consequence predictions, causal explanations, recommendation systems and topic-focussed exploration of scientific literature.