Exploring Explanations Improves the Robustness of In-Context Learning

TL;DR

This paper introduces X$^2$-ICL, an advanced explanation-based in-context learning framework that systematically explores explanations for all labels, significantly enhancing the robustness of large language models against out-of-distribution data.

Contribution

It extends existing explanation-based ICL methods by systematically exploring explanations for all labels, improving robustness and decision-making in language models.

Findings

X$^2$-ICL outperforms previous ICL methods on multiple datasets.

Significant robustness improvement to out-of-distribution data.

Enhanced understanding and articulation of reasoning processes.

Abstract

In-context learning (ICL) has emerged as a successful paradigm for leveraging large language models (LLMs). However, it often struggles to generalize beyond the distribution of the provided demonstrations. A recent advancement in enhancing robustness is ICL with explanations (X-ICL), which improves prediction reliability by guiding LLMs to understand and articulate the reasoning behind correct labels. Building on this approach, we introduce an advanced framework that extends X-ICL by systematically exploring explanations for all possible labels (X$^2$-ICL), thereby enabling more comprehensive and robust decision-making. Experimental results on multiple natural language understanding datasets validate the effectiveness of X$^2$-ICL, demonstrating significantly improved robustness to out-of-distribution data compared to the existing ICL approaches.