Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A

TL;DR

This paper introduces a new formalism and dataset for procedural knowledge, and proposes analogy-augmented generation (AAG) that leverages a memory store to improve procedural question answering, outperforming baselines.

Contribution

It presents a novel formalism and dataset for procedural knowledge, and introduces AAG, a method that uses analogy and memory to enhance procedural reasoning in language models.

Findings

AAG outperforms few-shot and RAG baselines on multiple datasets.

AAG shows superior performance in pairwise LLM evaluation.

Human evaluation confirms AAG's effectiveness in RecipeNLG.

Abstract

Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG.