Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry

TL;DR

This paper compares the generalization of dense text embeddings from LLMs and non-LLMs, showing how SLERP merging improves model robustness and preserves task-specific gains in embedding quality.

Contribution

It introduces a controlled experimental framework to evaluate embedding generalization and demonstrates that SLERP merging effectively mitigates over-specialization in LLM-based embeddings.

Findings

LLM-based embeddings better capture numeric patterns

SLERP merging restores base-model structure

Merged models outperform non-merged in clustering tasks

Abstract

We investigate the generalization properties of dense text embeddings when the embedding backbone is a large language model (LLM) versus when it is a non-LLM encoder, and we study the extent to which spherical linear interpolation (SLERP) model-merging mitigates over-specialization introduced by task-specific adaptation (e.g., LoRA). To make the comparison concrete and domain-agnostic, we design a controlled suite of experiments in which models embed short numerical sequences and are evaluated on their ability to cluster and classify those sequences according to well-defined number-theoretic properties. Our experimental protocol compares four families of models: (1) non-LLM encoders trained from scratch or fine-tuned for embeddings, (2) LLM-based encoders adapted with parameter-efficient methods (LoRA), (3) LLM-based encoders with LoRA followed by model souping merging into the base weights, and (4) the same LoRA-adapted LLMs merged using SLERP across checkpoints or stages. We evaluate representational quality with clustering indices (Silhouette and Davies Bouldin). We additionally analyze the use of kmeans labels to see if the embeddings encode any other information besides the one we are testing for. Empirically, we find that LLM-based backbones produce embeddings that better capture higher-order, compositional numeric patterns, but are prone to adapter dominance that degrades balanced generalization; SLERP merging consistently recovers base-model structure while retaining most task gains, yielding superior tradeoffs in clustering separability, and robustness compared to model souping or models that were not merged.