Disentangling semantics in language through VAEs and a certain architectural choice

TL;DR

This paper introduces an unsupervised approach using modified Transformers and VAEs to learn disentangled sentence representations, enabling manipulation of semantic elements like verbs and objects.

Contribution

It proposes a novel architecture that disentangles semantic components in sentences without supervision, facilitating targeted semantic modifications and swaps.

Findings

Successfully separates sentence elements into distinct latent variables

Varying latent variables alters specific sentence components

Swapping latent variables produces meaningful semantic changes

Abstract

We present an unsupervised method to obtain disentangled representations of sentences that single out semantic content. Using modified Transformers as building blocks, we train a Variational Autoencoder to translate the sentence to a fixed number of hierarchically structured latent variables. We study the influence of each latent variable in generation on the dependency structure of sentences, and on the predicate structure it yields when passed through an Open Information Extraction model. Our model could separate verbs, subjects, direct objects, and prepositional objects into latent variables we identified. We show that varying the corresponding latent variables results in varying these elements in sentences, and that swapping them between couples of sentences leads to the expected partial semantic swap.