E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

TL;DR

E2LLM introduces a novel method for long-context understanding in LLMs by compressing chunks of text into soft prompts, improving performance and efficiency in tasks like summarization and question answering.

Contribution

The paper proposes E2LLM, a new approach that effectively balances long-context performance, computational efficiency, and compatibility with pretrained models.

Findings

E2LLM outperforms 8 SOTA methods in effectiveness and efficiency.

Achieves top performance on LongBench v2 among comparable models.

Uses soft prompts and training objectives to enhance reasoning with long contexts.

Abstract

Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.