Shortcut Learning of Large Language Models in Natural Language Understanding

TL;DR

This paper reviews how large language models often rely on dataset biases as shortcuts, affecting their robustness, and discusses methods to identify, understand, and mitigate this shortcut learning behavior.

Contribution

It provides a comprehensive overview of recent methods to detect, analyze, and address shortcut learning in large language models, highlighting future research directions.

Findings

Identification methods for shortcut learning behaviors

Analysis of causes behind shortcut reliance

Discussion of mitigation strategies and challenges

Abstract

Large language models (LLMs) have achieved state-of-the-art performance on a series of natural language understanding tasks. However, these LLMs might rely on dataset bias and artifacts as shortcuts for prediction. This has significantly affected their generalizability and adversarial robustness. In this paper, we provide a review of recent developments that address the shortcut learning and robustness challenge of LLMs. We first introduce the concepts of shortcut learning of language models. We then introduce methods to identify shortcut learning behavior in language models, characterize the reasons for shortcut learning, as well as introduce mitigation solutions. Finally, we discuss key research challenges and potential research directions in order to advance the field of LLMs.