Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

TL;DR

This paper introduces ADVAE, a probabilistic model that learns to disentangle syntactic roles in sentence representations without supervision, enabling better interpretability and controllable content generation in NLP.

Contribution

The paper proposes ADVAE, an attention-based variational autoencoder that achieves unsupervised disentanglement of syntactic roles in sentence representations, outperforming classical models.

Findings

Disentanglement of syntactic roles achieved without supervision.

ADVAE outperforms classical sequence and Transformer VAEs in role separation.

Syntactic roles can be manipulated by intervening on latent variables.

Abstract

Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,$\dots$) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that $\textit{i)}$ disentanglement of syntactic roles can be induced without supervision, $\textit{ii)}$ ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, $\textit{iii)}$ realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available.