Explicit Inductive Inference using Large Language Models

TL;DR

This paper introduces a pipeline leveraging large language models' attestation bias to perform explicit inductive inference, improving inference accuracy and reducing bias effects.

Contribution

It proposes a novel pipeline that transforms premises into alternatives and aggregates answers to enhance LLM inference performance.

Findings

Improved LLM inference accuracy on a predicate entailment benchmark.

Significant reduction of attestation bias impact in inference tasks.

Demonstrated effectiveness of the pipeline across different LLMs.

Abstract

Large Language Models (LLMs) are reported to hold undesirable attestation bias on inference tasks: when asked to predict if a premise P entails a hypothesis H, instead of considering H's conditional truthfulness entailed by P, LLMs tend to use the out-of-context truth label of H as a fragile proxy. In this paper, we propose a pipeline that exploits this bias to do explicit inductive inference. Our pipeline uses an LLM to transform a premise into a set of attested alternatives, and then aggregate answers of the derived new entailment inquiries to support the original inference prediction. On a directional predicate entailment benchmark, we demonstrate that by applying this simple pipeline, we can improve the overall performance of LLMs on inference and substantially alleviate the impact of their attestation bias.