A Dialectic Pipeline for Improving LLM Robustness

TL;DR

This paper introduces a dialectic pipeline that enhances LLM robustness by enabling self-dialogue for answer reflection and correction, outperforming standard and Chain-of-Thought prompting across various datasets and models.

Contribution

It presents a novel self-dialogue based pipeline that improves LLM answer quality without sacrificing generalization, using context enrichment and reflection mechanisms.

Findings

Significant performance improvements over standard model answers.

Consistent outperformance compared to Chain-of-Thought prompting.

Effective across multiple datasets and model families.

Abstract

Assessing ways in which Language Models can reduce their hallucinations and improve the outputs' quality is crucial to ensure their large-scale use. However, methods such as fine-tuning on domain-specific data or the training of a separate \textit{ad hoc} verifier require demanding computational resources (not feasible for many user applications) and constrain the models to specific fields of knowledge. In this thesis, we propose a dialectic pipeline that preserves LLMs' generalization abilities while improving the quality of its answer via self-dialogue, enabling it to reflect upon and correct tentative wrong answers. We experimented with different pipeline settings, testing our proposed method on different datasets and on different families of models. All the pipeline stages are enriched with the relevant context (in an oracle-RAG setting) and a study on the impact of its summarization or its filtering is conducted. We find that our proposed dialectic pipeline is able to outperform by significative margins the standard model answers and that it consistently achieves higher performances than Chain-of-Thought only prompting.