DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs

TL;DR

This paper introduces the first large-scale benchmark for detecting hallucinations in Vietnamese LLMs, providing a dataset, challenge, and analysis to improve model reliability in low-resource language settings.

Contribution

It presents the ViHallu dataset and the DSC2025 ViHallu Challenge, establishing a new benchmark for hallucination detection in Vietnamese LLMs and analyzing diverse detection methods.

Findings

Best system achieved 84.80% macro-F1 score

Instruction-tuned LLMs outperform generic models

Detection of intrinsic hallucinations remains challenging

Abstract

The reliability of large language models (LLMs) in production environments remains significantly constrained by their propensity to generate hallucinations -- fluent, plausible-sounding outputs that contradict or fabricate information. While hallucination detection has recently emerged as a priority in English-centric benchmarks, low-to-medium resource languages such as Vietnamese remain inadequately covered by standardized evaluation frameworks. This paper introduces the DSC2025 ViHallu Challenge, the first large-scale shared task for detecting hallucinations in Vietnamese LLMs. We present the ViHallu dataset, comprising 10,000 annotated triplets of (context, prompt, response) samples systematically partitioned into three hallucination categories: no hallucination, intrinsic, and extrinsic hallucinations. The dataset incorporates three prompt types -- factual, noisy, and adversarial -- to stress-test model robustness. A total of 111 teams participated, with the best-performing system achieving a macro-F1 score of 84.80\%, compared to a baseline encoder-only score of 32.83\%, demonstrating that instruction-tuned LLMs with structured prompting and ensemble strategies substantially outperform generic architectures. However, the gap to perfect performance indicates that hallucination detection remains a challenging problem, particularly for intrinsic (contradiction-based) hallucinations. This work establishes a rigorous benchmark and explores a diverse range of detection methodologies, providing a foundation for future research into the trustworthiness and reliability of Vietnamese language AI systems.