Few-Shot Optimized Framework for Hallucination Detection in Resource-Limited NLP Systems

TL;DR

This paper presents a novel few-shot optimization framework that improves hallucination detection in resource-limited NLP systems by leveraging iterative prompt engineering, data restructuring, and fine-tuning of large language models, achieving state-of-the-art results.

Contribution

It introduces DeepSeek Few-shot optimization and a comprehensive framework for effective hallucination detection with limited data in NLP systems.

Findings

Achieved 85.5% accuracy on SHROOM task

Enhanced downstream model performance through data restructuring

Demonstrated effectiveness of few-shot optimization and fine-tuning

Abstract

Hallucination detection in text generation remains an ongoing struggle for natural language processing (NLP) systems, frequently resulting in unreliable outputs in applications such as machine translation and definition modeling. Existing methods struggle with data scarcity and the limitations of unlabeled datasets, as highlighted by the SHROOM shared task at SemEval-2024. In this work, we propose a novel framework to address these challenges, introducing DeepSeek Few-shot optimization to enhance weak label generation through iterative prompt engineering. We achieved high-quality annotations that considerably enhanced the performance of downstream models by restructuring data to align with instruct generative models. We further fine-tuned the Mistral-7B-Instruct-v0.3 model on these optimized annotations, enabling it to accurately detect hallucinations in resource-limited settings. Combining this fine-tuned model with ensemble learning strategies, our approach achieved 85.5% accuracy on the test set, setting a new benchmark for the SHROOM task. This study demonstrates the effectiveness of data restructuring, few-shot optimization, and fine-tuning in building scalable and robust hallucination detection frameworks for resource-constrained NLP systems.