CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models

TL;DR

This paper introduces CCHall, a new benchmark designed to evaluate large language models' ability to handle hallucinations across both cross-lingual and cross-modal scenarios, addressing a significant gap in current evaluation methods.

Contribution

The paper presents CCHall, the first benchmark to jointly assess cross-lingual and cross-modal hallucinations in LLMs, and provides comprehensive evaluation results highlighting current model limitations.

Findings

Current LLMs struggle with joint cross-lingual and cross-modal hallucinations.

CCHall serves as a valuable resource for future LLM evaluation.

Evaluation of mainstream open-source and closed-source LLMs on CCHall.

Abstract

Investigating hallucination issues in large language models (LLMs) within cross-lingual and cross-modal scenarios can greatly advance the large-scale deployment in real-world applications. Nevertheless, the current studies are limited to a single scenario, either cross-lingual or cross-modal, leaving a gap in the exploration of hallucinations in the joint cross-lingual and cross-modal scenarios. Motivated by this, we introduce a novel joint Cross-lingual and Cross-modal Hallucinations benchmark (CCHall) to fill this gap. Specifically, CCHall simultaneously incorporates both cross-lingual and cross-modal hallucination scenarios, which can be used to assess the cross-lingual and cross-modal capabilities of LLMs. Furthermore, we conduct a comprehensive evaluation on CCHall, exploring both mainstream open-source and closed-source LLMs. The experimental results highlight that current LLMs still struggle with CCHall. We hope CCHall can serve as a valuable resource to assess LLMs in joint cross-lingual and cross-modal scenarios.