Reducing Hallucination in Enterprise AI Workflows via Hybrid Utility Minimum Bayes Risk (HUMBR)

TL;DR

This paper introduces HUMBR, a hybrid utility approach to significantly reduce hallucinations in enterprise AI workflows by combining semantic and lexical methods, validated through extensive benchmarks and real-world deployment.

Contribution

We propose a novel Hybrid Utility MBR framework that synthesizes semantic and lexical techniques, providing rigorous error bounds and demonstrating superior performance over existing methods.

Findings

81% of suggestions were preferred over human ground truth

HUMBR significantly outperforms standard Universal Self-Consistency

Critical recall failures were virtually eliminated

Abstract

Although LLMs drive automation, it is critical to ensure immense consideration for high-stakes enterprise workflows such as those involving legal matters, risk management, and privacy compliance. For Meta, and other organizations like ours, a single hallucinated clause in such high stakes workflows risks material consequences. We show that by framing hallucination mitigation as a Minimum Bayes Risk (MBR) problem, we can dramatically reduce this risk. Specifically, we introduce a Hybrid Utility MBR (HUMBR) framework that synthesizes semantic embedding similarity with lexical precision to identify consensus without ground-truth references, for which we derive rigorous error bounds. We complement this theoretical analysis with a comprehensive empirical evaluation on widely-used public benchmark suites (TruthfulQA and LegalBench) and also real world data from Meta production deployment. The results from our empirical study show that MBR significantly outperforms standard Universal Self-Consistency. Notably, 81% of the pipeline's suggestions were preferred over human-crafted ground truth, and critical recall failures were virtually eliminated.