Deductive Additivity for Planning of Natural Language Proofs

TL;DR

This paper investigates whether embedding spaces can serve as efficient heuristics for planning in natural language proof generation by examining the property of deductive additivity across various embedding models.

Contribution

It introduces the concept of deductive additivity in embeddings and evaluates its effectiveness for planning in natural language reasoning tasks.

Findings

Standard embeddings often embed conclusions near the sum of premises

Embedding methods generally fall short as effective heuristics for deductive reasoning

Certain categories of reasoning are not well modeled by current embeddings

Abstract

Current natural language systems designed for multi-step claim validation typically operate in two phases: retrieve a set of relevant premise statements using heuristics (planning), then generate novel conclusions from those statements using a large language model (deduction). The planning step often requires expensive Transformer operations and does not scale to arbitrary numbers of premise statements. In this paper, we investigate whether an efficient planning heuristic is possible via embedding spaces compatible with deductive reasoning. Specifically, we evaluate whether embedding spaces exhibit a property we call deductive additivity: the sum of premise statement embeddings should be close to embeddings of conclusions based on those premises. We explore multiple sources of off-the-shelf dense embeddings in addition to fine-tuned embeddings from GPT3 and sparse embeddings from BM25. We study embedding models both intrinsically, evaluating whether the property of deductive additivity holds, and extrinsically, using them to assist planning in natural language proof generation. Lastly, we create a dataset, Single-Step Reasoning Contrast (SSRC), to further probe performance on various reasoning types. Our findings suggest that while standard embedding methods frequently embed conclusions near the sums of their premises, they fall short of being effective heuristics and lack the ability to model certain categories of reasoning.