Learning to Learn to be Right for the Right Reasons

TL;DR

This paper introduces a meta-learning approach to improve model generalization in commonsense reasoning by performing well on both easy and hard test sets, addressing superficial cue overfitting.

Contribution

It proposes a novel meta-learning method that explicitly trains models to perform well on both superficial and non-superficial test instances, enhancing robustness.

Findings

Up to 16.5 percentage points improvement over baseline.

Effective on COPA and Commonsense Explanation datasets.

Balances performance on easy and hard test sets.

Abstract

Improving model generalization on held-out data is one of the core objectives in commonsense reasoning. Recent work has shown that models trained on the dataset with superficial cues tend to perform well on the easy test set with superficial cues but perform poorly on the hard test set without superficial cues. Previous approaches have resorted to manual methods of encouraging models not to overfit to superficial cues. While some of the methods have improved performance on hard instances, they also lead to degraded performance on easy instances. Here, we propose to explicitly learn a model that does well on both the easy test set with superficial cues and hard test set without superficial cues. Using a meta-learning objective, we learn such a model that improves performance on both the easy test set and the hard test set. By evaluating our models on Choice of Plausible Alternatives (COPA) and Commonsense Explanation, we show that our proposed method leads to improved performance on both the easy test set and the hard test set upon which we observe up to 16.5 percentage points improvement over the baseline.