Extensible Embedding: A Flexible Multipler For LLM's Context Length

TL;DR

This paper introduces Extensible Embedding, a novel approach that enhances large language models by enabling flexible, high-quality context extension through a compact, information-dense embedding method that is cost-effective and compatible with existing models.

Contribution

The paper proposes Extensible Embedding, a new method that improves context extension in LLMs by representing multiple tokens as a single, information-rich embedding, enhancing flexibility and efficiency.

Findings

Supports diverse context lengths flexibly

Achieves high sample efficiency in training

Demonstrates superior performance in long-context tasks

Abstract

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we propose Extensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.