Contextual Lensing of Universal Sentence Representations

TL;DR

This paper introduces Contextual Lensing, a novel method for creating adaptable universal sentence vectors that can capture different language similarities, including translation, without requiring parallel data.

Contribution

It proposes a new framework for inducing context-dependent sentence embeddings using a core matrix and adaptable lens parameters, enhancing flexibility and multilingual capabilities.

Findings

Able to encode translation similarity across languages

Can focus language similarity notions with few lens parameters

Operates without parallel data for cross-lingual encoding

Abstract

What makes a universal sentence encoder universal? The notion of a generic encoder of text appears to be at odds with the inherent contextualization and non-permanence of language use in a dynamic world. However, mapping sentences into generic fixed-length vectors for downstream similarity and retrieval tasks has been fruitful, particularly for multilingual applications. How do we manage this dilemma? In this work we propose Contextual Lensing, a methodology for inducing context-oriented universal sentence vectors. We break the construction of universal sentence vectors into a core, variable length, sentence matrix representation equipped with an adaptable `lens' from which fixed-length vectors can be induced as a function of the lens context. We show that it is possible to focus notions of language similarity into a small number of lens parameters given a core universal matrix representation. For example, we demonstrate the ability to encode translation similarity of sentences across several languages into a single weight matrix, even when the core encoder has not seen parallel data.