Improving Robustness and Generality of NLP Models Using Disentangled Representations

TL;DR

This paper introduces a disentangled representation learning approach for NLP models, improving their robustness and domain generalization by mapping inputs to multiple independent representations and combining their predictions.

Contribution

The paper proposes novel methods to enhance NLP model robustness and generality through disentangled representations, including regularization techniques like L2 and Total Correlation within the variational information bottleneck framework.

Findings

Models with disentangled representations outperform baseline models in robustness.

Disentangled models show improved domain adaptation capabilities.

Proposed methods enhance stability against input perturbations.

Abstract

Supervised neural networks, which first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$, have achieved remarkable success in a wide range of natural language processing (NLP) tasks. Despite their success, neural models lack for both robustness and generality: small perturbations to inputs can result in absolutely different outputs; the performance of a model trained on one domain drops drastically when tested on another domain. In this paper, we present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. Instead of mapping $x$ to a single representation $z$, the proposed strategy maps $x$ to a set of representations $\{z_1,z_2,...,z_K\}$ while forcing them to be disentangled. These representations are then mapped to different logits $l$s, the ensemble of which is used to make the final prediction $y$. We propose different methods to incorporate this idea into currently widely-used models, including adding an $L$2 regularizer on $z$s or adding Total Correlation (TC) under the framework of variational information bottleneck (VIB). We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.