A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios

TL;DR

This paper introduces a new metric to evaluate the robustness of large language models against natural, non-malicious input variations, supported by empirical testing on specially constructed datasets.

Contribution

It proposes a novel robustness metric and demonstrates its effectiveness through empirical evaluation on datasets with natural perturbations.

Findings

The new metric effectively measures model robustness in non-adversarial scenarios.

Large language models show varying degrees of robustness depending on input perturbations.

Empirical results validate the usefulness of the proposed metric in real-world settings.

Abstract

We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.