FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models

TL;DR

FragRel enhances large language models' ability to process long texts by leveraging structural relations between text fragments in external memory, improving understanding and generation in complex, interconnected contexts.

Contribution

This work introduces a novel method to exploit fragment-level relations in external memory, enabling better handling of interconnected long texts in LLMs.

Findings

Improved long story understanding

Enhanced code generation from repositories

Better performance in long-term chatting

Abstract

To process contexts with unlimited length using Large Language Models (LLMs), recent studies explore hierarchically managing the long text. Only several text fragments are taken from the external memory and passed into the temporary working memory, i.e., LLM's context window. However, existing approaches isolatedly handle the text fragments without considering their structural connections, thereby suffering limited capability on texts with intensive inter-relations, e.g., coherent stories and code repositories. This work attempts to resolve this by exploiting the fragment-level relations in external memory. First, we formulate the fragment-level relations and present several instantiations for different text types. Next, we introduce a relation-aware fragment assessment criteria upon previous independent fragment assessment. Finally, we present the fragment-connected Hierarchical Memory based LLM. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.