Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder

TL;DR

This survey explores how autoencoder-based models can unify symbolic and distributional semantics by analyzing their latent space geometries to improve interpretability and semantic understanding.

Contribution

It introduces a novel perspective on latent space geometry in autoencoders for semantic representation learning, bridging symbolic and distributional semantics.

Findings

Compared VAE, VQVAE, and SAE in their latent geometries.

Highlighted how different autoencoders influence semantic interpretability.

Provided insights into enhancing semantic models with autoencoder architectures.

Abstract

Integrating compositional and symbolic properties into current distributional semantic spaces can enhance the interpretability, controllability, compositionality, and generalisation capabilities of Transformer-based auto-regressive language models (LMs). In this survey, we offer a novel perspective on latent space geometry through the lens of compositional semantics, a direction we refer to as \textit{semantic representation learning}. This direction enables a bridge between symbolic and distributional semantics, helping to mitigate the gap between them. We review and compare three mainstream autoencoder architectures-Variational AutoEncoder (VAE), Vector Quantised VAE (VQVAE), and Sparse AutoEncoder (SAE)-and examine the distinctive latent geometries they induce in relation to semantic structure and interpretability.