CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation

TL;DR

This paper describes CCNU's system for multilingual hallucination detection in question-answering, leveraging multiple LLMs with internal and external knowledge, achieving top results in several languages.

Contribution

The paper introduces a novel multi-LLM annotation approach that combines internal and external knowledge for multilingual hallucination detection.

Findings

Achieved top ranking in Hindi hallucination detection.

Secured Top-5 positions in seven other languages.

Provided insights into effective LLM-based annotation strategies.

Abstract

We present the system developed by the Central China Normal University (CCNU) team for the Mu-SHROOM shared task, which focuses on identifying hallucinations in question-answering systems across 14 different languages. Our approach leverages multiple Large Language Models (LLMs) with distinct areas of expertise, employing them in parallel to annotate hallucinations, effectively simulating a crowdsourcing annotation process. Furthermore, each LLM-based annotator integrates both internal and external knowledge related to the input during the annotation process. Using the open-source LLM DeepSeek-V3, our system achieves the top ranking (\#1) for Hindi data and secures a Top-5 position in seven other languages. In this paper, we also discuss unsuccessful approaches explored during our development process and share key insights gained from participating in this shared task.