Investigating the Effect of Natural Language Explanations on Out-of-Distribution Generalization in Few-shot NLI

TL;DR

This paper investigates whether natural language explanations can improve out-of-distribution generalization in few-shot NLI models, finding that generated explanations often hallucinate and do not enhance prediction accuracy.

Contribution

The study introduces a framework for testing natural language explanations' impact on OOD generalization in few-shot NLI, revealing limitations of current explanation generation methods.

Findings

Generated explanations often hallucinate information.

Explanations fail to improve OOD prediction performance.

BLEU scores of explanations are competitive with groundtruth, but do not correlate with accuracy.

Abstract

Although neural models have shown strong performance in datasets such as SNLI, they lack the ability to generalize out-of-distribution (OOD). In this work, we formulate a few-shot learning setup and examine the effects of natural language explanations on OOD generalization. We leverage the templates in the HANS dataset and construct templated natural language explanations for each template. Although generated explanations show competitive BLEU scores against groundtruth explanations, they fail to improve prediction performance. We further show that generated explanations often hallucinate information and miss key elements that indicate the label.