Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis

TL;DR

This paper investigates how different types of perturbations to exemplars in prompts affect the robustness of large language models in multi-hop reasoning, finding that certain perturbations decrease sensitivity and that more perturbed exemplars improve robustness.

Contribution

It introduces a domain-agnostic perturbation framework to systematically analyze LLM robustness in multi-hop reasoning tasks and shows that noisy exemplars enhance model robustness.

Findings

Models are sensitive to synonym replacements.

Increasing perturbed exemplars improves robustness.

Semantic perturbations impact model performance.

Abstract

Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.