Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy

TL;DR

This paper introduces a novel method to compare and visualize a large landscape of sentence encoders using Quantum Relative Entropy, enabling insights into their relationships and performance predictions.

Contribution

We develop a scalable approach to map and analyze over a thousand sentence encoders based on Quantum Relative Entropy, revealing their relationships and predicting downstream task performance.

Findings

The map accurately reflects relationships between encoders.

Encoder feature vectors can predict downstream task performance.

The method scales to over a thousand encoders.

Abstract

We propose a method to compare and visualise sentence encoders at scale by creating a map of encoders where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an embedding matrix of a sentence set, where each row corresponds to the embedding of a sentence. Next, we compute the Pairwise Inner Product (PIP) matrix for a sentence encoder using its embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its Quantum Relative Entropy (QRE) with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in retrieval and clustering tasks, demonstrating the faithfulness of our map.