Learning Disentangled Textual Representations via Statistical Measures of Similarity

TL;DR

This paper introduces a new family of regularizers based on statistical similarity measures that enable disentangled textual representations without additional training, improving efficiency and effectiveness over existing methods.

Contribution

The authors propose a novel regularization approach for disentangling sensitive attributes from text representations that does not require training or hyperparameter tuning.

Findings

Regularizers are faster and do not require training.

Achieves better disentanglement results with pretrained and random encoders.

Reduces complexity compared to adversarial and mutual information methods.

Abstract

When working with textual data, a natural application of disentangled representations is fair classification where the goal is to make predictions without being biased (or influenced) by sensitive attributes that may be present in the data (e.g., age, gender or race). Dominant approaches to disentangle a sensitive attribute from textual representations rely on learning simultaneously a penalization term that involves either an adversarial loss (e.g., a discriminator) or an information measure (e.g., mutual information). However, these methods require the training of a deep neural network with several parameter updates for each update of the representation model. As a matter of fact, the resulting nested optimization loop is both time consuming, adding complexity to the optimization dynamic, and requires a fine hyperparameter selection (e.g., learning rates, architecture). In this work, we introduce a family of regularizers for learning disentangled representations that do not require training. These regularizers are based on statistical measures of similarity between the conditional probability distributions with respect to the sensitive attributes. Our novel regularizers do not require additional training, are faster and do not involve additional tuning while achieving better results both when combined with pretrained and randomly initialized text encoders.