Methods for Estimating and Improving Robustness of Language Models

TL;DR

This paper reviews methods for estimating and improving the robustness of large language models, focusing on their generalization abilities and how incorporating certain measures can enhance their distributional robustness.

Contribution

It surveys diverse research approaches to estimate LLM generalization and demonstrates that integrating these measures into training improves robustness.

Findings

Incorporating generalization measures enhances model robustness.

Survey of diverse methods for estimating LLM robustness.

Future directions proposed for robustness improvement.

Abstract

Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions towards enhancing the robustness of LLMs.