Learning Disentangled Representations for Natural Language Definitions

TL;DR

This paper introduces a method using definitional sentences and semantic structures to train a Variational Autoencoder that learns disentangled representations, enhancing interpretability and downstream NLP tasks.

Contribution

It proposes a novel approach leveraging semantic regularities in definitional sentences to improve disentanglement in neural models for NLP.

Findings

Outperforms unsupervised baselines in disentanglement benchmarks.

Improves downstream definition modeling performance.

Utilizes semantic structures for better representation learning.

Abstract

Disentangling the encodings of neural models is a fundamental aspect for improving interpretability, semantic control and downstream task performance in Natural Language Processing. Currently, most disentanglement methods are unsupervised or rely on synthetic datasets with known generative factors. We argue that recurrent syntactic and semantic regularities in textual data can be used to provide the models with both structural biases and generative factors. We leverage the semantic structures present in a representative and semantically dense category of sentence types, definitional sentences, for training a Variational Autoencoder to learn disentangled representations. Our experimental results show that the proposed model outperforms unsupervised baselines on several qualitative and quantitative benchmarks for disentanglement, and it also improves the results in the downstream task of definition modeling.