Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets

TL;DR

This paper evaluates the challenges of unsupervised representation disentanglement in text using synthetic datasets, adapting models from image domain and highlighting key factors affecting disentanglement quality.

Contribution

It introduces the first framework and datasets for assessing unsupervised disentanglement in text, bridging a gap between image and text representation learning.

Findings

Disentanglement is more challenging in text than in images.

Representation sparsity influences disentanglement success.

Coupling with the decoder impacts disentanglement quality.

Abstract

To highlight the challenges of achieving representation disentanglement for text domain in an unsupervised setting, in this paper we select a representative set of successfully applied models from the image domain. We evaluate these models on 6 disentanglement metrics, as well as on downstream classification tasks and homotopy. To facilitate the evaluation, we propose two synthetic datasets with known generative factors. Our experiments highlight the existing gap in the text domain and illustrate that certain elements such as representation sparsity (as an inductive bias), or representation coupling with the decoder could impact disentanglement. To the best of our knowledge, our work is the first attempt on the intersection of unsupervised representation disentanglement and text, and provides the experimental framework and datasets for examining future developments in this direction.