An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks

TL;DR

This paper compares embedding algorithms from leading LLMs using medical data, analyzing similarity scores, implementing Siamese Neural Networks for enhancement, and evaluating their performance and carbon footprint.

Contribution

It provides a comparative analysis of embedding algorithms from top industry models and introduces Siamese Neural Networks to improve embedding performance.

Findings

Differences in similarity scores among algorithms

Siamese Neural Networks improved embedding accuracy

Carbon footprint varies across models and training epochs

Abstract

Large Lanugage Models (LLMs) are gaining increasing popularity in a variety of use cases, from language understanding and writing to assistance in application development. One of the most important aspects for optimal funcionality of LLMs is embedding layers. Word embeddings are distributed representations of words in a continuous vector space. In the context of LLMs, words or tokens from the input text are transformed into high-dimensional vectors using unique algorithms specific to the model. Our research examines the embedding algorithms from leading companies in the industry, such as OpenAI, Google's PaLM, and BERT. Using medical data, we have analyzed similarity scores of each embedding layer, observing differences in performance among each algorithm. To enhance each model and provide an additional encoding layer, we also implemented Siamese Neural Networks. After observing changes in performance with the addition of the model, we measured the carbon footage per epoch of training. The carbon footprint associated with large language models (LLMs) is a significant concern, and should be taken into consideration when selecting algorithms for a variety of use cases. Overall, our research compared the accuracy different, leading embedding algorithms and their carbon footage, allowing for a holistic review of each embedding algorithm.