OrLog: Resolving Complex Queries with LLMs and Probabilistic Reasoning

TL;DR

OrLog is a neuro-symbolic retrieval framework that combines LLM-based predicate plausibility estimation with probabilistic reasoning to improve complex query resolution with multiple constraints, outperforming existing methods.

Contribution

It introduces a decoupled approach for plausibility estimation and logical reasoning, enabling more reliable and efficient constraint-aware retrieval using LLMs.

Findings

Significantly improves top-rank precision on complex queries.

Reduces token usage by approximately 90% per query-entity pair.

Outperforms baseline retrievers and LLM-as-reasoner methods across various settings.

Abstract

Resolving complex information needs that come with multiple constraints should consider enforcing the logical operators encoded in the query (i.e., conjunction, disjunction, negation) on the candidate answer set. Current retrieval systems either ignore these constraints in neural embeddings or approximate them in a generative reasoning process that can be inconsistent and unreliable. Although well-suited to structured reasoning, existing neuro-symbolic approaches remain confined to formal logic or mathematics problems as they often assume unambiguous queries and access to complete evidence, conditions rarely met in information retrieval. To bridge this gap, we introduce OrLog, a neuro-symbolic retrieval framework that decouples predicate-level plausibility estimation from logical reasoning: a large language model (LLM) provides plausibility scores for atomic predicates in one decoding-free forward pass, from which a probabilistic reasoning engine derives the posterior probability of query satisfaction. We evaluate OrLog across multiple backbone LLMs, varying levels of access to external knowledge, and a range of logical constraints, and compare it against base retrievers and LLM-as-reasoner methods. Provided with entity descriptions, OrLog can significantly boost top-rank precision compared to LLM reasoning with larger gains on disjunctive queries. OrLog is also more efficient, cutting mean tokens by $\sim$90\% per query-entity pair. These results demonstrate that generation-free predicate plausibility estimation combined with probabilistic reasoning enables constraint-aware retrieval that outperforms monolithic reasoning while using far fewer tokens.