In Search for Linear Relations in Sentence Embedding Spaces

TL;DR

This paper investigates how small textual modifications in sentences are reflected in their vector representations within popular sentence embedding models, revealing that some embeddings encode these subtle differences.

Contribution

It provides an initial analysis of the relationship between minor sentence changes and their impact on continuous-space embeddings, highlighting the potential for linear relations.

Findings

Vector differences in some embeddings reflect small sentence changes

Small textual alterations can be captured by certain embedding models

The study offers insights into the structure of sentence embedding spaces

Abstract

We present an introductory investigation into continuous-space vector representations of sentences. We acquire pairs of very similar sentences differing only by a small alterations (such as change of a noun, adding an adjective, noun or punctuation) from datasets for natural language inference using a simple pattern method. We look into how such a small change within the sentence text affects its representation in the continuous space and how such alterations are reflected by some of the popular sentence embedding models. We found that vector differences of some embeddings actually reflect small changes within a sentence.