Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation

TL;DR

This paper introduces GSTransform, a lightweight framework that efficiently adapts pre-existing text embeddings to follow user instructions in real-time, significantly reducing computational costs while maintaining high embedding quality.

Contribution

GSTransform is a novel guided space transformation method that adapts generic embeddings to instruction-specific contexts without re-encoding the entire corpus.

Findings

Achieves 6-300x speedup in real-time embedding adaptation.

Improves instruction-following embedding quality over state-of-the-art methods.

Demonstrates effectiveness across multiple downstream tasks and datasets.

Abstract

In this work, we investigate an important task named instruction-following text embedding, which generates dynamic text embeddings that adapt to user instructions, highlighting specific attributes of text. Despite recent advancements, existing approaches suffer from significant computational overhead, as they require re-encoding the entire corpus for each new instruction. To address this challenge, we propose GSTransform, a novel instruction-following text embedding framework based on Guided Space Transformation. Our key observation is that instruction-relevant information is inherently encoded in generic embeddings but remains underutilized. Instead of repeatedly encoding the corpus for each instruction, GSTransform is a lightweight transformation mechanism that adapts pre-computed embeddings in real time to align with user instructions, guided by a small amount of text data with instruction-focused label annotation. We conduct extensive experiments on three instruction-awareness downstream tasks across nine real-world datasets, demonstrating that GSTransform improves instruction-following text embedding quality over state-of-the-art methods while achieving dramatic speedups of 6~300x in real-time processing on large-scale datasets. The source code is available at https://github.com/YingchaojieFeng/GSTransform.