UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Output

TL;DR

This paper presents a system for detecting and pinpointing hallucinations in large language model outputs across multiple languages, using context retrieval, false content identification, and prompt optimization, achieving top performance in SemEval 2025 Task 3.

Contribution

It introduces a novel framework combining context retrieval, false content detection, and prompt optimization for multilingual hallucination detection in LLMs, setting a new benchmark.

Findings

Achieved highest overall performance in SemEval-2025 Task 3

Effective multilingual hallucination detection across languages

Enhanced detection accuracy through prompt optimization

Abstract

Hallucinations pose a significant challenge for large language models when answering knowledge-intensive queries. As LLMs become more widely adopted, it is crucial not only to detect if hallucinations occur but also to pinpoint exactly where in the LLM output they occur. SemEval 2025 Task 3, Mu-SHROOM: Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes, is a recent effort in this direction. This paper describes the UCSC system submission to the shared Mu-SHROOM task. We introduce a framework that first retrieves relevant context, next identifies false content from the answer, and finally maps them back to spans in the LLM output. The process is further enhanced by automatically optimizing prompts. Our system achieves the highest overall performance, ranking #1 in average position across all languages. We release our code and experiment results.