Controllable Evidence Selection in Retrieval-Augmented Question Answering via Deterministic Utility Gating

TL;DR

This paper introduces a deterministic framework for selecting relevant evidence in retrieval-augmented question answering, improving evidence quality and transparency without requiring training or fine-tuning.

Contribution

It proposes Meaning-Utility Estimation and Diversity-Utility Estimation for explicit, rule-based evidence selection prior to answer generation, enhancing transparency and control.

Findings

Produces compact, auditable evidence sets

Eliminates the need for training or fine-tuning

Ensures evidence explicitly satisfies task requirements

Abstract

Many modern AI question-answering systems convert text into vectors and retrieve the closest matches to a user question. While effective for topical similarity, similarity scores alone do not explain why some retrieved text can serve as evidence while other equally similar text cannot. When many candidates receive similar scores, systems may select sentences that are redundant, incomplete, or address different conditions than the question requires. This paper presents a deterministic evidence selection framework for retrieval-augmented question answering. The approach introduces Meaning-Utility Estimation (MUE) and Diversity-Utility Estimation (DUE), fixed scoring and redundancy-control procedures that determine evidence admissibility prior to answer generation. Each sentence or record is evaluated independently using explicit signals for semantic relatedness, term coverage, conceptual distinctiveness, and redundancy. No training or fine-tuning is required. In the prototype, a unit is accepted only if it explicitly states the fact, rule, or condition required by the task. Units are not merged or expanded. If no unit independently satisfies the requirement, the system returns no answer. This deterministic gating produces compact, auditable evidence sets and establishes a clear boundary between relevant text and usable evidence.