Learning Invariant Representations for Sentiment Analysis: The Missing Material is Datasets

TL;DR

This paper introduces metrics and a data filtering method to evaluate and improve the robustness of sentiment analysis models against nuisance factors like product ID, highlighting the importance of dataset annotation for invariant representation learning.

Contribution

It proposes two generalization metrics and a simple filtering approach to assess and mitigate nuisance factors in NLP sentiment analysis datasets.

Findings

Nuisance factors can significantly bias sentiment classification.

Filtering data based on nuisance annotations improves model robustness.

The approach is applicable to datasets with annotated nuisance variables.

Abstract

Learning representations which remain invariant to a nuisance factor has a great interest in Domain Adaptation, Transfer Learning, and Fair Machine Learning. Finding such representations becomes highly challenging in NLP tasks since the nuisance factor is entangled in a raw text. To our knowledge, a major issue is also that only few NLP datasets allow assessing the impact of such factor. In this paper, we introduce two generalization metrics to assess model robustness to a nuisance factor: \textit{generalization under target bias} and \textit{generalization onto unknown}. We combine those metrics with a simple data filtering approach to control the impact of the nuisance factor on the data and thus to build experimental biased datasets. We apply our method to standard datasets of the literature (\textit{Amazon} and \textit{Yelp}). Our work shows that a simple text classification baseline (i.e., sentiment analysis on reviews) may be badly affected by the \textit{product ID} (considered as a nuisance factor) when learning the polarity of a review. The method proposed is generic and applicable as soon as the nuisance variable is annotated in the dataset.

Figures