
TL;DR
This paper introduces a novel family of item embeddings based on Information Geometry, specifically the $oldsymbol{ extit{ extalpha}}$-geometry of the exponential family, providing a unified framework that includes standard embeddings like Word2Vec and GloVe.
Contribution
It develops a new theoretical framework for item embeddings using $ extit{ extalpha}$-geometry, offering insights into how this deformation affects embedding quality and evaluation.
Findings
$ extit{ extalpha}$-parameter influences embedding performance
Unified geometric framework for various embedding methods
Analysis of $ extit{ extalpha}$-deformation effects on NLP tasks
Abstract
Learning an embedding for a large collection of items is a popular approach to overcome the computational limitations associated to one-hot encodings. The aim of item embedding is to learn a low dimensional space for the representations, able to capture with its geometry relevant features or relationships for the data at hand. This can be achieved for example by exploiting adjacencies among items in large sets of unlabelled data. In this paper we interpret in an Information Geometric framework the item embeddings obtained from conditional models. By exploiting the -geometry of the exponential family, first introduced by Amari, we introduce a family of natural -embeddings represented by vectors in the tangent space of the probability simplex, which includes as a special case standard approaches available in the literature. A typical example is given by word embeddings,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsGloVe Embeddings
