Zero-Shot Belief: A Hard Problem for LLMs

TL;DR

This paper investigates the challenge of zero-shot belief prediction using large language models, proposing two approaches and demonstrating their effectiveness on FactBank and ModaFact datasets.

Contribution

It introduces a unified and a hybrid LLM-based system for zero-shot belief prediction, achieving state-of-the-art results and providing detailed error analysis.

Findings

Hybrid approach achieves new state-of-the-art on FactBank.

LLMs struggle with zero-shot belief prediction tasks.

Error analysis reveals key challenges in the task.

Abstract

We present two LLM-based approaches to zero-shot source-and-target belief prediction on FactBank: a unified system that identifies events, sources, and belief labels in a single pass, and a hybrid approach that uses a fine-tuned DeBERTa tagger for event detection. We show that multiple open-sourced, closed-source, and reasoning-based LLMs struggle with the task. Using the hybrid approach, we achieve new state-of-the-art results on FactBank and offer a detailed error analysis. Our approach is then tested on the Italian belief corpus ModaFact.