Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models

TL;DR

This paper constructs a dataset to evaluate LLM robustness against noisy external information and proposes data augmentation and contrastive learning methods to improve model resilience and discrimination capabilities.

Contribution

It introduces a novel dataset simulating interference scenarios and proposes a fine-tuning approach with contrastive learning to enhance LLM robustness against noise.

Findings

Improved model robustness against noisy information

Enhanced discrimination capability of LLMs

Validated effectiveness using GPT-4 evaluations

Abstract

The development of LLMs has greatly enhanced the intelligence and fluency of question answering, while the emergence of retrieval enhancement has enabled models to better utilize external information. However, the presence of noise and errors in retrieved information poses challenges to the robustness of LLMs. In this work, to evaluate the model's performance under multiple interferences, we first construct a dataset based on machine reading comprehension datasets simulating various scenarios, including critical information absence, noise, and conflicts. To address the issue of model accuracy decline caused by noisy external information, we propose a data augmentation-based fine-tuning method to enhance LLM's robustness against noise. Additionally, contrastive learning approach is utilized to preserve the model's discrimination capability of external information. We have conducted experiments on both existing LLMs and our approach, the results are evaluated by GPT-4, which indicates that our proposed methods improve model robustness while strengthening the model's discrimination capability.