End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering

TL;DR

This paper presents an automatic, scalable evaluation framework for chatbots that leverages LLMs for response judgment and confidence filtering, reducing human effort and applicable across languages and domains.

Contribution

The authors introduce a modular, language-agnostic evaluation system that automates chatbot assessment using LLMs and confidence filtering, improving scalability and reducing manual review.

Findings

High agreement with human judgments on Vietnamese news dataset

Significantly reduces review overhead

Modular and adaptable to various domains

Abstract

Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q\&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention.