End-to-End Self-Debiasing Framework for Robust NLU Training

TL;DR

This paper presents a simple end-to-end debiasing framework for NLU models that improves out-of-distribution robustness while maintaining in-distribution accuracy by jointly training a bias model with the main model.

Contribution

The authors propose a novel, simple joint training framework that leverages shallow representations to derive a bias model, enhancing OOD performance in NLU tasks.

Findings

Significantly improves OOD performance across three NLU tasks.

Outperforms existing debiasing methods on two tasks.

Maintains high in-distribution accuracy.

Abstract

Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones. We introduce a simple yet effective debiasing framework whereby the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously. We demonstrate on three well studied NLU tasks that despite its simplicity, our method leads to competitive OOD results. It significantly outperforms other debiasing approaches on two tasks, while still delivering high in-distribution performance.