Talk Less, Verify More: Improving LLM Assistants with Semantic Checks and Execution Feedback

TL;DR

This paper presents two verification techniques, Q* and Feedback+, that enhance the accuracy and reliability of large language model assistants in enterprise workflows by incorporating semantic checks and execution feedback.

Contribution

It introduces a novel generator-discriminator framework with verification mechanisms that improve LLM output correctness and trustworthiness in business analytics tasks.

Findings

Q* reduces semantic errors in code generation.

Feedback+ improves code correctness through execution feedback.

Both methods decrease task completion time and error rates.

Abstract

As large language model (LLM) assistants become increasingly integrated into enterprise workflows, their ability to generate accurate, semantically aligned, and executable outputs is critical. However, current conversational business analytics (CBA) systems often lack built-in verification mechanisms, leaving users to manually validate potentially flawed results. This paper introduces two complementary verification techniques: Q*, which performs reverse translation and semantic matching between code and user intent, and Feedback+, which incorporates execution feedback to guide code refinement. Embedded within a generator-discriminator framework, these mechanisms shift validation responsibilities from users to the system. Evaluations on three benchmark datasets, Spider, Bird, and GSM8K, demonstrate that both Q* and Feedback+ reduce error rates and task completion time. The study also identifies reverse translation as a key bottleneck, highlighting opportunities for future improvement. Overall, this work contributes a design-oriented framework for building more reliable, enterprise-grade GenAI systems capable of trustworthy decision support.