RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models

TL;DR

RLIE is a framework that combines large language models with probabilistic rule learning, involving rule generation, weight learning, iterative refinement, and evaluation, to improve neuro-symbolic reasoning accuracy.

Contribution

RLIE introduces a novel integrated approach that couples LLM-generated rules with probabilistic modeling and iterative refinement for enhanced reasoning.

Findings

Directly applying weighted rules improves performance.

Prompting LLMs with rules and weights can reduce accuracy.

LMMs are better at semantic tasks than probabilistic integration.

Abstract

Large Language Models (LLMs) can propose rules in natural language, sidestepping the need for a predefined predicate space in traditional rule learning. Yet many LLM-based approaches ignore interactions among rules, and the opportunity to couple LLMs with probabilistic rule learning for robust inference remains underexplored. We present RLIE, a unified framework that integrates LLMs with probabilistic modeling to learn a set of weighted rules. RLIE has four stages: (1) Rule generation, where an LLM proposes and filters candidates; (2) Logistic regression, which learns probabilistic weights for global selection and calibration; (3) Iterative refinement, which updates the rule set using prediction errors; and (4) Evaluation, which compares the weighted rule set as a direct classifier with methods that inject rules into an LLM. We evaluate multiple inference strategies on real-world datasets. Applying rules directly with their learned weights yields superior performance, whereas prompting LLMs with the rules, weights, and logistic-model outputs surprisingly degrades accuracy. This supports the view that LLMs excel at semantic generation and interpretation but are less reliable for precise probabilistic integration. RLIE clarifies the potential and limitations of LLMs for inductive reasoning and couples them with classic probabilistic rule combination methods to enable more reliable neuro-symbolic reasoning.